Hi all.
I want to develop a plugin system within my program, and I have a trait that functions defined by plugins should implement.
Currently, my code gets all the functions in a HashMap and then calls them by their name. Problem is, I have to create that hashmap myself by inserting every function myself.
I would really appreciate it if there was a way to say, suppose, all pub members of mod functions:: that implement this trait PluginFunction call register(hashmap) function. So as I add more functions as mod in functions it’ll be automatically added on compile.
Pseudocode:
Files:
src/
├── attrs.rs
├── functions
│ ├── attrs.rs
│ ├── export.rs
│ └── render.rs
├── functions.rs
├── lib.rs
Basically, in mod functions I want:
impl AllFunctions{
pub fn new() -> Self {
let mut functions_map = HashMap::new();[[
register_all!(crate::functions::* implementing PluginFunction, &mut functions_map);
Self { function_map }
}
}
Right now I’m doing:
impl AllFunctions{
pub fn new() -> Self {
let mut functions_map = HashMap::new();[[
crate::functions::attrs::PrintAttr{}.register(&mut functions_map);
crate::functions::export::ExportCSV{}.register(&mut functions_map);
crate::functions::render::RenderText{}.register(&mut functions_map);
// More as I add more functions
Self { function_map }
}
}
How about parsing the source files?
Getting metadata like this requires adding an extension to the language, either directly in code (as with a macro) or by messing with the build process (running another pass through your code to extract the metadata).
If you’re not adverse to using a global, you could write a macro that you use instead of
implfor these traits. So, for example:register_plugin!(impl MyTrait for MyStruct { /* regular impl stuff goes here */ });This macro would populate the global with metadata about the association between MyStruct and MyTrait. You could then pass a copy of it to
AllFunctions::new.Alternatively, write a macro that just does the registration, which is like what you’re already doing:
impl MyTrait for MyStruct { ... } register_plugin!(MyTrait, MyStruct);Another option would be to write something funky in
build.rsthat does whateverrustdocdoes to discover impls and pass that to the rest of your code as const data. As a hack, you could have it invokerustdocand then parse the output.How dynamic is this plugin system, though? If you can only change the registered plugins by rebuilding, then automatic discovery doesn’t seem so warranted. Invoking a simple registration macro at the bottom of each file would save a lot of complexity budget.
Thank you for your detailed response.
I am ok using macros. But even proc macro only get the tokens and using in on the whole mod is unstable unless you use use it on
mod sth{...}instead of code being on in a different file (sth.rs).The plug-in system is dynamic in a sense that my plans for it are loading them through shared libraries (.dll, .so) compiled separately by users. But I also have internally provided core plugins that come with the program. But rust ABI system is not that stable, so in worst case I might have to ask users to just add plugin code to some directory and re-compile program instead of loading from shared libraries. That’s why I’m trying to make it as simple as possible. Asking users to modify the rust code somewhere else yo register the plugin might be met with resistance.
I was thinking that using build script to parse the source code and generating those codes could work, but that seemed hacky. So I was trying to see if there are better solutions, as it felt like a problem people might have come across themselves.
You could wrap the entirety of your file in a monster macro but you’d still have to assign the macro result to a variable you need to register, which doesn’t sound viable to me at least.
Maybe you can use a script that would extract all the trait implementations and create the boilerplate glue code for you, something like this:
grep --recursive --only-matching "impl PluginFunction for \w*" functions/ | sed --quiet "s/functions\/\(.*\)\.rs:impl PluginFunction for \(\w*\)/crate::functions::\1::\2{}.register(\&mut functions_map)/p"I tried to recreate your situation locally but it may not match perfectly, maybe you’ll have to adjust it a little. When I run it on my file tree which looks like this
functions ├── attr.rs ├── export.rs └── render.rs 1 directory, 3 fileswhere every file has a content like this
// comment pub struct MyAttrStructName {} impl PluginFunction for MyAttrStructName { }Then I receive the following output:
crate::functions::attr::MyAttrStructName{}.register(&mut functions_map) crate::functions::export::MyExportStructName{}.register(&mut functions_map) crate::functions::render::MyRenderStructName{}.register(&mut functions_map)Thank you. Yeah, something like this would work for me as I can add in a script and run it before compiling. But it won’t be a cross platform solution and windows/mac users are probably not going to be able to do anything. Maybe if I do the same thing but from build.rs it’ll work. I’ll try that.
No, macros can see only the tokens you give them. They have no notion of the fact that
crate::functionsis a module, thatPluginFunctionis a trait and thatfunctions_mapis a variable. Not even the compiler may know those informations when the macro is expanded.If you really really want to do this, you can use something like
inventory. Note thatinventoryuses a special linker section to do this, which some consider a hack. This is also not supported on WASM if you want to target it.Thank you. I just put the call with
!, I don’t necessarily want a macro solution. Any solution is acceptable, my requirement is that I can just keep adding more mods with functions insrc/functions/and not have to register each function.Inventory seems like the solution I am looking for. Although in my case, instead of collecting different values of the same type, I want to collect different types that all have same trait. But maybe I can make a temporary struct with
Box<dyn _>member to collect it if trying to collect it directly doesn’t work. I do not plan to support WASM. I am planning to make C/C++ and Python API for the libraries though, so if it has problems with them, then I might have a problem.Maybe keep maintaining the HashMap you have now and use one of these less portable mechanisms in a test to alert you when you forgot to register one?
That seems like a good compromise if I don’t find something better. Thank you.
I’m hoping to make it easy for people to add more functions, that’s why I want minimal code change required to add more functions.