1

For a while now I have been toying with the idea of using source code as a file storage format.

My question: How to support format version changes? (loading of older files with structural differences)

An example would be a settings file.

JSON:

{
  "version": 1,
  "email":   "[email protected]"
}

Then there is typically a manual or automatic parsing/deserialization/binding step that maps the textual information to a typed model in memory, e.g. an object of type Settings:

public class Settings {
  public String email;
}

Using source code as storage format, the settings file would look like this:

//version=1
public class MySettingsFile1 {
  public static Settings load() {
    Settings s = new Settings();
    s.email = "[email protected]";
    return s;
  }
}

The settings file could then be loaded by leveraging the class loading mechanism:

String className = "MySettingsFile1";
String source = ... (read settings file to string)
Class<?> clazz = new CustomClassLoader().compileAndLoad(className, source);
Settings settings = (Settings) clazz.getMethod("load").invoke(null);
System.out.println(settings.email);

I already know out how to load a class from a String.

Pros

  • no parsing / binding code required
  • text format (human readable)
  • concise (verbosity comparable to JSON)
  • supports comments

Cons

  • how to support different versions and upgrade from older versions?
  • security (sandbox the loading code?)

Edit: The real use case I have in mind are files that are only modified by a custom GUI editor, and then compiled and packaged with the application (no security concerns here). These files are already under version control (XML currently).

Improve this question
5
  • 1
    security should be your main concern. By definition, data is "owned" by the end user. It sounds like a nightmare to security that you want to execute that. Commented May 19, 2022 at 8:13
  • @tofro I agree, the settings file was a bad example. My actual use case are files that would not be edited by the end user, but packaged with the application. Commented May 19, 2022 at 8:19
  • The "packaged with the application" case is rather different; no need for a custom class loader, you can just build it into the application. This is not unusual in the C world where a chunk of constants or binary blob may be encoded as an array and compiled. Commented May 19, 2022 at 9:39
  • 1
    Always the downvotes from this community. This guy has a valid question and there is a valid answer. Commented May 19, 2022 at 17:32
  • GitLab does this - their configuration files are actually Ruby files, the same language most of GitLab is written in. The software is open core, and the company is quite open with their documentation. Commented May 31, 2022 at 15:53

3 Answers 3

Reset to default
5

With the question clarified as to handling differences in the structure of configuration from one version to the next, you lose a lot of potential solutions by using a code as configuration process. There are several factors that are problematic:

  • Code as configuration is inherently more brittle as it is much harder to isolate the configuration abstraction from the rest of the application
  • If you are not careful, something that works on one version of your software will simply cause your system to crash due to hidden state in the configuration objects
  • Most serialization code allows you to name the configurable attributes separately from the bound objects, allowing the config structure to remain unchanged
  • Some file formats have well known tools to transform one schema to another. For example, XSLT or JSON transformations would allow you to convert legacy schemas to newer schemas easily

In order to pull of code as configuration, you need the following in place:

  • Rigorous separation of config from application. Leaky abstractions can break your system.
  • Dynamic loading (not all languages have this, on purpose), as well as not trusting the code that is loaded.
  • Even more robust error handling.
    • Need to handle logical errors before everything is configured
    • Need to ensure proper order of initializing code from the configuration
    • Need to provide clear and accurate troubleshooting information in error handling
    • Need to prevent illegal actions
  • Clearly defined behavior when malicious code is injected into the configuration

You are kidding yourself if you think having a GUI to perform the configuration and compile the results is going to save yourself from malicious code. Viruses are insidious things, and it's really easy to get security wrong. You are much safer using serialization code that has lots of eyes on it than you are custom and proprietary formats--particularly binary.

I recommend against this approach for the following reasons:

  • It only works for interpreted (like shell scripts) or dynamically compiled languages (like Python)
  • Adding compilation for other languages (like Java, C, C#) adds more complexity than the binding provides
  • It is inherently insecure, allowing malicious users to inject their own code in a configuration file
  • You require people who may not have knowledge of programming languages to learn it just so they can configure your service
  • Describing how to configure something becomes even more complicated

There is a reason that file formats like JSON, YAML, XML, and others exist. It exists because it only represents data and not logic. You want the separation of responsibilities so that those who configure a service only have to worry about the settings they need to change.

To answer the question about version control of configuration files, I recommend you use a tool that is designed for that purpose.... like Git. For example, it is common for GitOps practitioners to have their configurations in a separate repository so that the changes can be applied to the different environments. To make that a reality, you need a continuous deployment tool (CD) that pulls the configs from your repository. At that point, rolling back is no different than simply rolling back changes in the configuration repository.

Improve this answer
4
  • Thank you for your input. Sorry, my question is not about version control, but how to handle renaming or structural changes in the data, and still be able to load old files. Commented May 19, 2022 at 6:49
  • How does that change just because configuration is source? I would argue the problem becomes harder with code as config because you don't have the option of translating an older config format to a newer one. Commented May 19, 2022 at 20:05
  • I've added additional information. In my job I have to where many hats, and both my engineering and security senses are tingling with the hidden dangers in the approach you are pursuing. Commented May 19, 2022 at 20:29
  • @BerinLoritsch and yet, as I mentioned in a comment on the question, GitLab does this. Theirs is probably a slightly different use case, security wise, as having access to a configuration file on someone's git server means hope is already lost. Commented May 31, 2022 at 15:57
2

The other answer has already covered the main point why storing configuration in source is a bad idea. But I wanted to add a few points.

Just about every developer should know how json works, and there are great libraries and tools for just about every platform. Using any custom format will make the system more difficult to use for anyone familiar with standard development practices.

How can you be certain that there are no security concerns? Have you considered all possible ways to use the software, and any possible future application? There are lots of example security problems caused by the developer making assumptions that does not hold up in the real world.

To handle changes in format you should have a layer specifically for handling data. Smaller changes, like adding or renaming properties can usually be accommodated by configuring your parser. Larger structural changes would be made by checking the format version of the file, and use the appropriate parsing for that version. After parsing is done you would have converters that convert the data the current representation. But significant data format changes should be rare, especially for something like configuration.

There could be application when storing settings in source could be useful. For example if a customer fills in some details, and a build system makes some changes and creates a customer specific build. But this would be a very specialized system, not something used for regular configuration.

Improve this answer
1
  • Thank you for your input. Like I said, I'm just toying with the idea, exploring. I know how to handle storage format migrations in conventional scenarios (XML/JSON/SQL). I'm interested in ideas for how to handle this when the file format is source code. Commented May 19, 2022 at 10:02
1

I do disagree with some of the other answers that seem to generally dispose of the idea that configuring through code could be a thing - After all, that's the general idea behind Lua, which is quite widespread as a configuration language, especially in the gaming industry. Unfortunately, Lua is not well prepared to embed into Java code (although there is, or used to be, a LuaJava project)

Embedding configuration in code (or using a script-like config language) opens up some interesting possibilities, like configuring through expressions that can be evaluated at run time or using control structures, like loops, for repeating configuration data.

Now, let's see what's different between the Lua approach and the one you propose:

  1. The Lua interpreter runs in an isolated sandbox, tightly controlled by the embedding program. What a Lua program can or cannot do with the data of the main program is controlled through host program code and the extent is thus clearly defined. Your application code needs to actively expose anything it wants exposed to the Lua interpreter - When embedding Java into Java, that's not the case (other than loading the "configuration" into a separate interpreter instance, which sort of defeats the point of doing it in the first place). Once you load a java class file, it inadvertently has complete access to all of your global code and data. There's no in-built method that would, for example, prevent malicious "configuration" from starting a separate thread that pushes all your private data to the Internet...
  2. Sandboxing Java in Java is hard (maybe even impossible), and it's actually very easy to inadvertently create loopholes for malicious code. Even such simple things just as preventing code to access the file system and delete all your user's files would be hard.
  3. Java classes need to be compiled. That's a major pain when trying to fix configuration in the field (which happens to be a more frequent thing that you may think) - You need to have the JDK installed on the target hosts which is basically an unnecessary nuisance, especially considering Oracle's current licence policies.

Scripting configuration has some very interesting and powerful traits to it, but using a compiled, full-blown language like Java to do it seems to be on the one hand add a lot of complexity in handling, and on the other hand open up severe security problems. I would go for something more lightweight than Java.

With regards to your actual question (how to handle versioning) I don't see why there should be any difference in handling configuration code and your normal application code. As long as you consider that code as part of your application (which you seem to do, as you've stated that configuration would not be subject to end-user modifications), code that handles internal configuration should be handled just like everything else. I can't see any "new" rules to apply that weren't valid to, for example, exposing an API.

Improve this answer
1
  • Thank you, I didn't know about Lua or that it is used for configuration in the gaming industry. I agree with the security concerns. Sorry, my question is not about version control, but how to handle renaming or structural changes in the data, and still be able to load old files. Commented May 31, 2022 at 8:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.