Lua file scraper (for mods)

Need to build a utility that scrapes the Lua game data and builds it into the JSON format this app expects. This would be the first step towards supporting mods.

https://www.reddit.com/r/factorio/comments/gje3t1/factorio_lab/fql2kyz?utm_source=share&utm_medium=web2x

I recently forked a calculator from KirkMcDonald, the goal was to rewrite all the code on Go in pure Lua. Result — https://bitbucket.org/hhrhhr/factorio-tools-lua
Since Lua 5.3 was used, some mods that actively use modified Lua 5.2 cannot be loaded yet (for example, angelsrefining_0.11.10). However, the rest of the mods are imported without problems, more precisely, they work the same way as in the original factorio-web-calc.
The developers shared the source code of modified Lua, so I can update this repository. I can also try to provide data.raw and a file with icons in the format you need.

Here is the test result from vanilla. Obtained completely automatically from game files, the code for using most mods is almost ready. There are some rough edges (like displaying empty / full barrels), but all the data and values are correct.
In the near future I will publish the source code of the parser, but maybe there will already be some comments?
For my part, I want to note that the calculator really needs the ability to load external data (textures with sprites, a css file and, of course, a data file).

0-18-25_css.json.txt
0-18-25.json.txt

P. S.
Аnd here I tried to "render" how sprites will be arranged using the sort order from the game.

So, I represent a completely (almost) workable parser - https://bitbucket.org/hhrhhr/factorio-lab-tools
The downloads have a ready-made set of all the libraries in the archive (unfortunately, only for Windows x64). In fact, GNU/Linux requires Lua, lua-zzip, lua-gd, and several other modules in pure Lua.
After starting the utility, the following files will appear in the current directory:

0-18.html, 0-18.css — demo page. rename css it to icons.scss (and delete the first few lines with demo ... classes)
0-18.json, * 0-18.png — need to copy to the appropriate directories

There are several difficulties.
I don’t know yet how to use the available tools to “color” the icons of the barrels of liquids (now they are all the same).
I also had to forcibly remove the "burner-mining-drill", since it consumes coal, which it itself produces, which leads to a loop and an error. I don’t know yet, is this the result of incorrect export or a feature of Factorio Lab :)
I enclose ready-made files from 0.18.26. if everything is fine with them, it remains only to connect the processing of mods (in archives and unpacked).
icons.scss.txt
0-18.json.txt

Thanks for your effort! I haven't had the time to work on this very much in the past few days, I'll check it out asap.
Calculations for burner mining drills are extremely complicated. You end up in a loop where you might be using the drills to produce coal but consume solid fuel, then if you're using coal liquefaction that also requires coal but also the added complexity of oil recipes. In Factorio Lab itself I don't allow using burner mining drills to produce raw ores, I only allow electric mining drills. This is generally ok (and the Kirk McDonald calculator has the same limitation) because it's extremely unusual to use burner mining drills at the stage you would be using a calculator like this.

I don’t know yet how to use the available tools to “color” the icons of the barrels of liquids (now they are all the same).

I maintain a different data export tool which can already do this correctly (but doesn't do some other things that yours does). The tool is written in Java, so code will be different, but maybe it helps to see how another tool does it: https://github.com/demodude4u/Java-Factorio-Data-Wrapper/blob/master/FactorioDataWrapper/src/com/demod/factorio/FactorioData.java#L121-L124 + https://github.com/demodude4u/Java-Factorio-Data-Wrapper/blob/master/FactorioDataWrapper/src/com/demod/factorio/Utils.java#L290-L308

Btw, if anyone is having trouble dealing with some of the more complicated mods feel free to reach out to me. I helped update Foreman to support the larger mods like Bobs. There a few things that are hard to anticipate...
Foreman: https://github.com/Rybadour/Foreman

So I've been kicking this idea in my head lately... Since there are now a few different tools that use item and recipe data from mods. We should try to focus all our attention on solidifying this scaper tool. One addition to that is instead of us parsing mods at runtime we could also ask mod owners to run this tool to generate a JSON file they publish along side the mod.

@Rybadour, now you have me imagining a scenario where mods could even build-in an SVG representation of the icons as part of the JSON itself, rendering the sprite sheets unnecessary. And if there were a hosted repository of these mod JSON files, apps like Factorio Lab could run against those at runtime rather than requiring them to be built in. Imagine if mods.factorio.com had a public API that could return the types of items, recipes, and icons that the mods added.

Yes! Maybe we can ask the Factorio devs for this. It would save us a ton of effort...

64px version from 0.18.27 ;)
icons.scss.txt

@Rybadour,

we could also ask mod owners to run this tool to generate a JSON file they publish along side the mod.

Some mods have their own settings. For example, you can choose to use water in recipes or change the color of the ore to blue.
In this case, you need to ask the authors of the mods to generate all possible variants of their configurations, which will lead to a power-law increase in the number of these configurations depending on the number of settings...
Therefore, I believe that "parsing mods at runtime" is indispensable.

I mean if some items are conditional then that could be incorporated into the tool that generates these settings. Another mechanism could be to generate the 99% that are static and a separate file that describes the configurable 1% or the describes all that encompasses the 1% variants.

@hhrhhr I think you are most likely correct, but obviously it'd be much nicer if we could handle mods at runtime instead of build time. In my opinion the best case is a hybrid where you have to use a tool like yours to generate files for your mod set, then you can load those into the app at runtime instead of build time. That's what I plan to work towards for now.

Big update. Added parser support for mods (so far only unpacked), icon painting (using tint), support for arbitrary scaling (48px, for example), the ability to disable mod loading or texture generation.
Examples (as on the top image) — https://hhrhhr.bitbucket.io/factorio_lab_tools/
Issue: tthe technology tree is not scanned; all possible recipes and products are displayed.
Sorry, but the endless technology and the new json file format are still at work ;) But the resulting data.raw almost perfectly matches what can be pulled out by the mod when loading the game. I saw the difference only in the 8-10th digit of numbers with double precision.

Current planned strategy for mod support:

mods folder inside assets folder
Subfolder for each built in supported mod
Supported mods provide json with items, recipes, and icons for the mod, same basic schema as current json
When support is requested for a new or updated mod, new branch/fork is created by me or by community, and a tool like @hhrhhr 's is used to generate new files and tested for support
Provide a guide on how to build new files/support for mods
When ready, fork/branch is merged into master and deployed to live site
UI will provide options to enable/disable supported mods

Thoughts?

Sounds like you don't plan to read mods dynamically? Foreman has had great success reading the mods directly from the mod directory. The biggest mods only take several seconds to parse everything.

I would love to but I don't see how it could be done. The browser doesn't have permission to directly read files from the local machine, and doesn't have lua language support. If you know of a way then I agree it would be far better to read dynamically.

I think you could load the json file via a request and then consider a setting that lets you provide a new json file. The json file would need to change slightly to contain a url to the sprite file but otherwise it should be able to load everything at runtime.
Obviously all the compile time dependencies to the json file would need to be removed (certainly non-trivial)...

Where would this mod data be hosted? As far as I know no mod data like this currently exists. If there were a public REST API for mods that'd be simple but I think we have to start at square one.

Github pages seems like a good enough place to me to host these files. Some of them could even be static assets in the https://factoriolab.github.io site published as part of the build.
Anybody who wants to load a mod could put a repo with a json file and image up and then put the url to the json file in a setting.
And you are correct it doesn't exist right now.

I think we are largely on the same track, then. The plan I outlined just exclusively has the files hosted inside factoriolab.github.io. In order to read from other sources either Factorio Lab would need to recognize those sources or there would need to be a way to specify external sources in the UI, which is feasible, just doesn't seem as user friendly.

Ah sorry I misunderstood. I thought we were talking about a script the user runs on their computer to generate mod data as a JSON file. I guess that's one advantage Foreman has if we want to parse mods we simply find the mod directory and extract some zip files.
So FYI, working with Foreman has taught me mods are very complicated. For example, many of their Lua only work when they can import other mods they depend on. Some imports modules from core or base so those would have to be available as well.

I thought we were talking about a script the user runs on their computer to generate mod data as a JSON file.

To be fair, yes, that is what I'm talking about. However, I don't intend that every user has to do that, only if you are using a mod that is not yet supported. In that case the user would run a script, generate the files, and create a PR to merge them into the repository, and after that the next person that wants to use that mod should be able to use the built in files.
Using generated JSON at runtime should help us avoid Lua confusion. If we can generate the files, then we have a manual step to make sure they're valid, then they have to go through a PR before they're available for anyone to use, then the tool to generate the data files doesn't have to be 100% perfect.
Dependencies will be complicated to handle, if a recipe from one mod requires ingredients from another mod it will either have to automatically enable the other mod, or will have to be handled like wood where there is no recognized recipe for the item until the dependency mod is enabled as well.

The next update (https://bitbucket.org/hhrhhr/factorio-lab-tools/src/master/). Added:

localization choice;
the ability to "clear" data.raw from unused values (greatly minimizes size);
a separate key for hacks only for Factorio Lab (pass burner-mining-drill as a producer, for example);
almost the correct order of loading mods (independent from each other mods may not load exactly as in the game);

Adding endless technologies so far is considered useless, since the right amount of scientific packages can be added to the calculation manually. However, space science packages, I believe, will be added later; so far, instead of 1000 of these packages, it is possible to use 100 Rocket parts (launch cost) as a temporary replacement.
Unfortunately, the resulting .json, although displayed one-to-one as in the game interface, in the calculator only works for simple recipes (without the use of oil and uranium). Apparently, I do not fully understand the format of the json file, in which the items group contains both the items themselves and links to their recipes. Because of this, fill-crude-oil-barrel recipes (for example) that produce a crude-oil-barrel product (with a different name) are simply not understood by the calculator.
And the crude-oil itself can be obtained both by extraction from the field, and using the empty-crude-oil-barrel, which further confuses the calculator.
This is one of the simplest examples, in some mods most of the resources can be obtained by disassembling objects containing them, that is, ordinary iron plates have almost hundreds of options for obtaining them.  And I don’t quite remember exactly where, but someone used the possibility of using a catalyst (as in the Kovarex process) in general for any ore in their recipes.
In general, until these details are clarified, I will temporarily suspend work for the exporter. Using other values for variables in .json or adding new ones is not a problem, just to know which ones;)
But in any case, I love this calculator only for the ability to select assembly machines directly in the general list, choosing the type of machine and combination of modules to achieve fewer machines or consumed resources (but still I need a calculation of energy consumption).

And a little suggestion on the .json file format.

If you need support for localizations, then they must be in separate files. There are a lot of supported languages in Factorio and keeping dozens of copies of the same config is not entirely rational.
Further, with the support of different files for the icons, you have to duplicate the same file name in each record.

  "icons": [ {
    { "file": "assets/0.18/0.18.31_lab.png", "id": "wooden-chest", "position": "0px 0px", "size": 64 },
    { "file": "assets/0.18/0.18.31_lab.png", "id": "iron-chest", "position": "-64px 0px", "size": 64 },
...
    { "file": "assets/0.18/0.18.31_category.png", "id": "logistics", "position": "-896px -768px", "size": 128 }.
    { "file": "assets/0.18/0.18.31_category.png", "id": "production", "position": "-960px -768px", "size": 128 }.

More compact would look like this (and generate separate images for different sizes of icons a little easier ;) :
  "icons": [ {
    { "file": "assets/0.18/0.18.31.png", "size": 64, "params": {
        { "id": "wooden-chest", "position": "0px 0px" },
        { "iron-chest", "position": "-64px 0px" },
...
    }, { "file": "assets/0.18/0.18.31_category.png", "size": 128, "params": {
        { "id": "logistics", "position": "-896px -768px" },
        { "id": "production", "position": "-960px -768px" },
...
    }

The position property can be written simply as an array, "position": [-960, -768], giving the calculation of the necessary values to the js-script.

Because of this, fill-crude-oil-barrel recipes (for example) that produce a crude-oil-barrel product (with a different name) are simply not understood by the calculator.

I probably need to improve this logic somewhat. Generally speaking, the calculator expects if there is a single recipe for an item, the recipe id should be the same as the item id. There should likely be a fallback that checks for simple recipes that only output the desired item.

If you need support for localizations, then they must be in separate files.

Localization hasn't been requested at this point, I do agree this would have to be kept separate.

with the support of different files for the icons, you have to duplicate the same file name

Your suggestion makes sense here, I can likely update the implementation.

The position property can be written simply as an array

How much does this matter? The current implementation makes it incredibly easy for the app to render the icons, your update would require a bit extra processing. It wouldn't require very much effort, I'm assuming this would make generating the mod json noticeably simpler?
@hhrhhr As a general note, some things should work a lot better now, the calculator should better handle modded complex recipes and the old oil and uranium utilities are completely gone. I don't expect it to be completely compatible with your modded json yet but it's getting there.

@hhrhhr Could I get some clarification on how to build your tool? I set the value for CC to my path to tcc.exe, and when I run mass_make_tcc.cmd, all I get in output is Press any key to continue . . ., and as best I can tell nothing is built.
Should I be able to run the .cmd directly from a Windows command prompt?

if all ok, then the output should be like this (for Lua 5.3):
build build_lua_53...
build build_zlib...
build build_zziplib...
build build_libpng...
build build_libgd...
build build_luafilesystem...
build build_lua_zlib...
build build_luazip...
build build_lua_gd...
build build_md5...
build build_luasocket...
build mime...
build build_serpent...
build build_binaryheap...
build build_timerwheel...
build build_copas...
build build_xavante...
build build_json...
all done

If this is not the case, then change set V=0 to set V=1 and look for errors or warnings.

error: include file 'unistd.h' not found

probably you did not install (did not unpack) winapi-full-for-0.9.27

probably you did not install (did not unpack) winapi-full-for-0.9.27

Looks like you're right, I missed that.
I might have a branch for you to play with soon that will work better. The only things I see wrong with your example output currently are:

A bunch of the recipes (like the barrel filling) are listed as items
The raw inputs like ores are listed as "other" instead of intermediate products as they are in game
Space science packs seem to be totally missing, no icon or anything, does this have to do with the fact there is no real recipe for them?

My in progress branch will let you pick a base mod (like 0.17, 0.16, 0.18, or overhaul mods like krastorio) as well as optional mods that only add a small number of items/recipes. I moved the infinite science recipes into an optional mod in this new branch, so there's no need for you to parse those in your tool.

raw inputs like ores are listed as "other" instead of intermediate products as they are in game

Аny ore have the type "item", a subgroup of "raw-resource" and a group of "intermediate-products". But ore (and also liquids) cannot be crafted, only mined. Ores and liquids do not have “recipes” and, in fact, are precisely the raw resources.
Therefore, it seems to me that ore and liquids should not be displayed in the calculator at all except for the ingredients (the "other" section should not be shown at all).

...(like the barrel filling) are listed as items

This is because at the time the parser was created, the "item" name and the final product in the recipe should have been the same.
But:

recipe
ingredient
product

fill-crude-oil-barrel
crude-oil, empty-barrel
crude-oil-barrel

empty-crude-oil-barrel
crude-oil-barrel
cude-oil, empty-barrel

and the calculator threw an exception because it could not find an item named fill-crude-oil-barrel or empty-crude-oil-barrel. And also the problem is that this type of recipes has several output products.

Space science packs...

...have no recipe at all like all other products in the game. They are the only representative of the rocket_launch_product property, which is a member of the only satellite product that is manufactured in the only rocket-silo "factory" from the hidden rocket-part product ("hidden" means that this product cannot be obtained, thrown away, picked up or put in a chest).
In other words, this recipe should be hardcoded manually, which I tried to avoid as much as possible so that the parser could work with absolutely any version of the game or mods.
Just in case, let me remind that the parser starts with a list of technologies that open a list of available recipes. Therefore, things like "player-port", "* -loader", "railgun"... do not fall into the final list of recipes; in the json file, this is the "recipes" section. Then the “items” section is created from all the unique ingredients and products from the recipes. After that, all the involved icons are retrieved. Plus, mods can change or turn off both technologies and recipes, which is also taken into account by the parser.

in progress branch...

I see only master branch. Maybe you will give an example of how the finished json-file should look on the example of:

items

crude-oil
empty-barrel
crude-oil-barrel

recipes

fill-crude-oil-barrel
empty-crude-oil-barrel
empty-barrel

Therefore, it seems to me that ore and liquids should not be displayed in the calculator at all except for the ingredients (the "other" section should not be shown at all).

I've often used a calculator to determine how many drills I need to fill a belt with ores, though. If we blocked ores out this wouldn't be possible. The intention of the item picker is to mirror the item picker in-game, as much as possible. I've also tested many times with outputs of petroleum, light oil, lubricant, sulfuric acid, etc. I'd prefer we just expose the items as they are in game, and the user will know best what they want to use.

at the time the parser was created, the "item" name and the final product in the recipe should have been the same.

That makes sense, that issue has been fixed though.

Space science packs...

If I could make a suggestion, I would add a flag to allow including them (hardcoded) when the tool is run, where the flag defaults to false. I definitely want them included for base versions of the game and adding them manually after running the tool is awkward because the sprite sheet would also need to be updated.

I see only master branch

My apologies. I've pushed the branch now, it's here, and updated basic game data is here. To answer your questions:

items

crude-oil

{
  "id": "crude-oil",
  "name": "Crude Oil",
  "category": "intermediate-products",
  "row": 1
}

empty-barrel

{
  "id": "empty-barrel",
  "name": "Empty Barrel",
  "stack": 50,
  "category": "intermediate-products",
  "row": 5
}

crude-oil-barrel

{
  "id": "crude-oil-barrel",
  "name": "Crude Oil Barrel",
  "stack": 50,
  "category": "intermediate-products",
  "row": 4
}

recipes

fill-crude-oil-barrel

{
  "id": "fill-crude-oil-barrel",
  "time": 0.2,
  "in": { "empty-barrel": 1, "crude-oil": 50 },
  "out": { "crude-oil-barrel": 1 },
  "producers": ["assembling-machine-2", "assembling-machine-3"]
}

empty-crude-oil-barrel

I haven't included this recipe in my data, partly because I don't think the empty barrel recipes should ever be used by this calculator, but please point out if you see a situation where they might be used

empty-barrel

{
  "id": "empty-barrel",
  "time": 1,
  "in": { "steel-plate": 1 },
  "producers": [
    "assembling-machine-1",
    "assembling-machine-2",
    "assembling-machine-3"
  ]
}

As part of this latest branch, I would like to start generating the core raw data from your tool, and include data for 0.16, 0.17, and 0.18. (Otherwise the UI to pick a version of the game is pretty silly)

probably you did not install (did not unpack) winapi-full-for-0.9.27

While you were correct, I've tried unzipping that file into the tcc folder in various different ways and encountered various different errors. I can't find any documentation indicating what I'm supposed to actually do with that file and its contents. I've tried copying the include folder into tcc\include, that resulted in errors. I've tried copying the orig folder into tcc\include and then copying the include folder into tcc\include, that resulted in different errors.
The main error I get is:
.../tcc/include/unistd.h:24: error: ',' expected (got "useconds")
It looks like my sys/types.h does not have a definition for useconds_t

The include directory from the archive must be unpacked with overwriting the existing directory %TCC_DIR%/include.
orig directory - a backup copy from tcc-0.9.27-win64-bin.zip\tcc\include.

error: ',' expected (got "useconds")

Yes, I forgot about the patch for %TCC_DIR%\include\unistd.h:
- int __cdecl __MINGW_NOTHROW usleep(useconds_t useconds);
+ int __cdecl __MINGW_NOTHROW usleep(useconds_t);

Second solution — just comment #define Z_HAVE_UNISTD_H in factorio-lab-tools\src\zlib\zconf.h.

Thanks for the comments, I'll try those tweaks to see if I can get it to work.

--factorio_lab_hacks
I noticed this! I don't think the existing hacks are necessary any more, though. I removed the mining productivity icon from settings and I have a separate "mod" for the infinite sciences, so I don't think it's necessary that you try to add the infinite technologies to your parser either. Also I haven't really tried it but I think the burner-mining-drill should work now, it shouldn't be necessary to remove it any more.

@hhrhhr Now I get tcc: error: undefined symbol 'floorf'
I see src/libgd/floorf.c and its comment that TCC has trouble with floorf, but I'm not sure what I'm supposed to do with it.

floorf.c:
#if defined(__TINYC__) && defined(__i386)
As I remember, the TCC 32 bit version has difficulties with the floorf function, that's why the macro is used. But for the 64-bit version there shouldn't be such difficulties. What version of tcc do you have (tcc.exe -v)?

C:\Program Files\tcc>tcc -v
tcc version 0.9.27 (x86_64 Windows)

I tried commenting out the #if defined and #endif in that file to force it to use that definition, and it got past that error, and then I ran into:
=> "C:\Program Files\tcc\tcc.exe" -shared -o C:\Git\factorio-lab-tools\dist53\lib\lua\5.3\lfs.dll -IC:\Git\factorio-lab-tools\dist53\include lfs.c -LC:\Git\factorio-lab-tools\dist53\bin -llua53
tcc: error: undefined symbol 'CreateSymbolicLinkA'
tcc: error: undefined symbol 'CreateHardLinkA'
tcc: error: undefined symbol 'GetFinalPathNameByHandleA'

Please, show output of git log -1.

C:\Git\factorio-lab-tools>git log -1
commit a34b297b77f89aceaa1820fab1794d9fce25264b (HEAD -> master, origin/master, origin/HEAD)
Author: Dmitry Zaitsev<EMAIL_ADDRESS>Date:   Sat Jun 13 23:02:42 2020 +0000

    * readme about kirkmcdonald fork

Should we move this to an email thread or Discord/other chat thread?

ok, try running the command to create and update TCC .def files.
cd /d C:\Program Files\tcc\lib
..\tcc.exe -impdef kernel32.dll

It looks like that did the trick! It still doesn't work unless I comment out the #if defined line in floorf.c, though. Thanks for your help!

Looks like tcc.exe -impdef msvcrt.dll also needed (at least floorf is present in msvcrt.dll in Win10 2004.)

@hhrhhr It's mostly working now, but the sprite sheet is filled in with black instead of icons. Any idea what I might be missing or doing wrong?
C:\Git\factorio-lab-tools\lua_lab>..\dist53\bin\lua53 factorio_data_dump.lua export -v -g "C:\Program Files (x86)\Steam\steamapps\common\Factorio"
[NOTE] args = {
  browse = false,
  clear = false,
  debug = false,
  export = true,
  factorio_lab_hacks = false,
  gamedir = "C:/Program Files (x86)/Steam/steamapps/common/Factorio",
  iconsize = 32,
  is_windows = true,
  language = "en",
  moddir = "C:/Users/dcbroad3/AppData/Roaming/Factorio/mods",
  noimage = false,
  nomods = false,
  suffix = "",
  verbose = 2
}
[INFO] mods settins loaded
[INFO] mods load order received:
        core <- base <- Bottleneck
[INFO] load order applied
[INFO] core and data load...
[NOTE]  loaded core.dataloader
[NOTE]  loaded core.data
[NOTE]  loaded base.data
[NOTE]  loaded base.data-updates
[INFO] exporting data.raw...
[INFO] process all items... found 162 with type="item", 78 with other type
[INFO] process technologies... enabled 181 recipes
[INFO] process recipes... found 206 normal, 18 expensive, 8 hidden
[NOTE]
hidden recipes: fast-loader, railgun, railgun-dart, electric-energy-interface, small-plane, express-loader, player-port, loader,
[INFO] sort recipes...
[INFO]  found 88 ingridients and 196 products
[INFO] clear items...
[NOTE]  23 ingridients and 15 products left
[NOTE]
sorted items: wood, coal, stone, iron-ore, copper-ore, uranium-ore, solid-fuel, crude-oil-barrel, heavy-oil-barrel, light-oil-barrel, lubricant-barrel, petroleum-gas-barrel, sulfuric-acid-barrel, water-barrel, uranium-235, uranium-238, used-up-uranium-fuel-cell, water, crude-oil, steam, heavy-oil, light-oil, petroleum-gas
[INFO] icon's canvas 16x15 (|xx| show number of layers)...
row   1 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row   2 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row   3 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row   4 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row   5 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row   6 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row   7 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 4|
row   8 | 4| 4| 4| 4| 4| 4| 4| 4| 4| 4| 4| 4| 4| 1| 1| 1|
row   9 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row  10 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row  11 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row  12 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row  13 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row  14 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 3| 3| 3| 3| 3| 3|
row  15 | 3| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
[INFO]
all done

I only had this in the case of using 32 bit TCC. For some reason the libgd library in 32-bit version is buggy (only in TCC, not GCC) when using the gd.copyResampled function on images with transparency, but when using the 64 bit version of TCC everything was fine.
Try edit ...\lua_lab\factorio\gd_image.lua, lines 73-74, comment canvas:copyResampled and uncomment canvas:copyResized. There will be a small loss in quality.

I'm not sure I understand your comments. I'm using the 64 bit version of TCC, are you saying it should work fine with that version? With which version should it work without this edit that causes a loss in quality?

I will try again to describe the compilation and launch process.

git clone https://bitbucket.org/hhrhhr/factorio-lab-tools.git
note: check git settings for auto converting LF/CRLF
let TCC_DIR=e:\tmp\tcc
unpack tcc-0.9.27-win64-bin.zip\tcc*.* to %TCC_DIR%
unpack winapi-full-for-0.9.27.zip\winapi-full-for-0.9.27\include\*.* to %TCC_DIR%\include (with overwrite)
execute commands:

cd /d %TCC_DIR%\lib
..\tcc.exe -impdef kernel32.dll
..\tcc.exe -impdef user32.dll
..\tcc.exe -impdef msvcrt.dll
..\tcc.exe -impdef ws2_32.dll

edit mass_make_tcc.cmd:
set CC=%TCC_DIR%\tcc.exe

for Lua 5.3:

set DIST=%R%dist53
set LUAVER=53

for Lua 5.2f (from Factorio team):

set DIST=%R%dist52f
set LUAVER=52f

run mass_make_tcc.cmd
cd lua_lab and run command:

for Lua 5.3
..\dist53\bin\lua53.exe factorio_data_dump.lua --gamedir %FACTORIO_DIR% --iconsize 24 --nomods --clear --suffix _53 export
for Lua 5.2f
..\dist52f\lua52f.exe factorio_data_dump.lua --gamedir %FACTORIO_DIR% --iconsize 24 --nomods --clear --suffix _52f export

in both case output looks like (for 0.18.36):
[INFO] mods settins loaded
[INFO] mods load order received
[INFO] load order applied
[INFO] core and data load...
[INFO] exporting data.raw...
[INFO] process all items... found 162 with type="item", 78 with other type
[INFO] process technologies... enabled 181 recipes
[INFO] process recipes... found 206 normal, 18 expensive, 8 hidden
[INFO] sort recipes...
[INFO]  found 88 ingridients and 196 products
[INFO] clear items...
[INFO] icon's canvas 16x15 (|xx| show number of layers)...
-- cutted --
[INFO]
all done

And texture looks ok:

I attach an archive with current sources and ready-made binaries. The lua_lab directory contains the resulting 0.18.36_*.(json|png). Compare them with your own, at the same time you can check all binaries for matches (TCC always makes binary-repeatable builds).
hhrhhr-factorio-lab-tools-211cd90f3991.zip

So I've been able to get it working by switching from copyResampled to copyResized.  I checked back through your directions and as best I can tell I've followed them precisely. Using copyResized it does seem like my sprite sheet is not precisely the same as yours, the file is smaller.

Did you use binaries from my archive and still get a "black" picture?
The "resize" filter, when reducing the image by N times, simply copies every N pixel. "resample" analyzes the surrounding pixels and calculates an average result. Therefore, in the second case, a smoothed picture is obtained, which is less amenable to compression (especially the alpha channel).

Binaries from your archive seem to work fine, binaries built locally using mass_make_tcc seem to create the black icons. It seems like I should be able to just use the binaries from your archive, I suppose.

If the sources and compiler are exactly the same, then the difference can only be in the system dynamic library msvcrt.dll. Although I also have no idea what exactly this difference could be :((

I'm not 100% sure what I did differently, but I got it working as expected this morning. A few things I cleaned up: I had to re-add the lcc directory to my PATH (apparently it was missing), I was able to rebuild using mass_make_tcc without any local modifications (previously I'd had to tweak floorf.c). Then, the export worked without modifying gd_image, so it's working using copyResampled!
Minor update: It seemed bad to me that your app had to include the time and empty module icons just to get my app to work. I've refactored so that those icons are provided directly by the app no matter what (as well as all the other icons that are explicitly used, mostly in the settings panel).
I'm going to try to see if I can build data for 0.16, 0.17, and 0.18 using your tool and include that in my latest branch. Main thing is that I want to get it to include the space-science-pack.

After updating the def files for TCC, the hacks for floorf are no longer needed. It would be necessary to remove them altogether from the source, somehow I will do it.
About previous versions of the game. I don't remember exactly, but they used slightly different calculations for the speed and volume of conveyors and something else.
t.belt = belt and {speed = 480.0 * belt.speed}
It will be necessary to somehow determine the version of the game.

they used slightly different calculations for the speed and volume of conveyors

That's true, I hadn't really considered that. I think that was true in 0.16 and hasn't been true since 0.17+ though, so a manual tweak to the numbers for belts in the 0.16 json shouldn't be a major issue, as long as it's not necessary to support a wide variety of 0.16 mods as well.

@hhrhhr Thanks for all of your help so far, Factorio Lab now supports 0.16, 0.17, and 0.18 thanks to you!
I did attempt to take the next step and start to include mods, but it seems as though mods are not loaded by your tool? I went all in and tried Krastorio 2, it threw an error regarding a missing dependency, I ignored it and it rebuilt normal 0.18 base data. I tried doing the simplest possible and simply used base game with the waterfill mod, I no longer got any errors about missing dependencies, but it still didn't load the data from the mod. Do you expect it to work with mods yet?

Can you show "error regarding a missing dependency"?

Krastorio is now exported seemingly without errors (apart from those that are present in the game's log).
When processing images, ignore warnings like:
libpng warning: Interlace handling should be turned on when using png_read_image
This is due to the deprecated png-library in Adobe ImageReady that the author of the mod used.

So I synced with your changes and I still don't see any mod data generated when I run the tool. I tried again with both Waterfill and Krastorio 2, in both cases the output is the base 0.18 game data with no mods. With Krastorio 2:
C:\Git\factorio-lab-tools\lua_lab>..\dist53\bin\lua53 factorio_data_dump.lua --gamedir "%factorio%" --iconsize 64 --clear export
[INFO] mods settins loaded
[INFO] mods load order received
[INFO] load order applied
[INFO] core and data load...
[INFO] exporting data.raw...
[INFO] process all items... found 162 with type="item", 78 with other type
[INFO] process technologies... enabled 181 recipes
[INFO] process recipes... found 199 normal, 18 expensive, 15 hidden
[INFO] sort recipes...
[INFO]  found 82 ingridients and 195 products
[INFO] clear items...
[INFO] icon's canvas 16x15 (|xx| show number of layers)...
row   1 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row   2 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row   3 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row   4 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row   5 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row   6 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row   7 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 4|
row   8 | 4| 4| 4| 4| 4| 4| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row   9 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row  10 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row  11 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row  12 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row  13 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row  14 | 1| 1| 3| 3| 3| 3| 3| 3| 3| 1| 1| 1| 1| 1| 1| 1|
row  15 | 1| 1| 1| 1|
[INFO]
all done

I do have some local tweaks to factorio_data_export.lua, but I tried reverting everything and running the scripts from your master branch against Krastorio 2, and I still don't get any mod data in the output.
Is it possible you tested against Krastorio and not Krastorio 2? I haven't tried with Krastorio since it's not compatible with 0.18.

Only unpacked modifications are supported.

So, you need unpack Krastorio2_0.9.99.zip to mods directory or in any another directory ( in this case set option --moddir and copy mod-list.json and mod-settings.dat too).
...
[INFO] mods settins loaded
[INFO] mods load order received:
        core <- base <- Krastorio2
[INFO] load order applied
[INFO] core and data load...
[NOTE]  loaded core.dataloader
[NOTE]  loaded core.data
[NOTE]  loaded base.data
[NOTE]  loaded Krastorio2.data
[NOTE]  loaded base.data-updates
[NOTE]  loaded Krastorio2.data-updates
[LOG]   Krastorio 2|LogLevel:WARNING
[LOG]   From Krastorio 2->file:..._u/Krastorio2_0.9.99/lib/private/data-stages/recipes.lua->function:findNotUnlockableRecipes->line:1883
[LOG]   Called in->file:...anilla-changes/optional/modify-science-packs-recipes.lua->function:func->line:17
[LOG]   Message: 'This recipes are not unlockable, maybe is an error: { "uranium-rounds-magazine", "nuclear-fuel", "imersite-rounds-magazine" } [from findNotUnlockableRecipes()]'
[NOTE]  loaded Krastorio2.data-final-fixes
...

Ohh, OK. Sorry I'm blundering around here, I will try that the next chance I get.

I'm making fairly steady progress with the Krastorio 2 mod, largely the issues I've been encountering is for producers in Krastorio, there are multiple unexpected combinations of group and subgroup (smelting producers that are actually assembling machines, assembling machines that are actually furnaces). I'm currently stuck on item names, as it seems loc_name isn't filled in for many items (it's using the entity name instead of the item name, and entity name looks more like a coded id).
I think many of the recipes are also going to pose problems because there are many that, for example, output "0" of a "void product". But I haven't gotten to those yet.
If you have any ideas on that I'd love to hear it, otherwise I'll keep playing with it later.

I've added fix to handle strings like __ITEM__kr-medium-container__: https://bitbucket.org/hhrhhr/factorio-lab-tools/commits/517679df3cbf69306b034ea095695df0c96150cb
Recipes with zero result, as I understand it, apply to some kind of machine that heats liquid. but I could not determine what these machines are. Can it be simple to exclude such "zero" recipes?

Аdded a json based demo page for 0.18.36 + Krastorio2, I think it will be a little more clearer.
https://hhrhhr.bitbucket.io/factorio_lab_tools/0.18.36_krastorio2.html

That's pretty cool! Do you know what the deal is with the smelting-crafting category? Everything under that tab also appears under intermediate products, and when I ran the app, that tab ended up showing as empty (I don't currently support showing things in two tabs).
Would it make more sense for me to convert the "row" number field in the data to something like "subgroup", since that's a more direct correlation to what's in the data? I don't think it would be a major effort.
I haven't tried out your fixes for the __ITEM__kr-medium-container__ string names yet, but your demo page still seems to show the incorrect names.
I was also thinking it might be easiest to simply exclude the zero-result recipes. The end result is they show something like:
"out": {
  "kr-void": 0
}

It doesn't make any sense to me to include recipes like that.

Recipes from group smelting-crafting shown in kr-advanced-assembling-machine (Advanced assembling machine):

And, judging by the product ID, these are different products from the point of view of the game. Using a "Iron beam" as an example (quoted from data.raw):
    ["kr-s-c-iron-beam"] = {
      type = "recipe",
      name = "kr-s-c-iron-beam",
      subgroup = "smelting-crafting",
      hide_from_player_crafting = true, -- <-- !!!
-- skipped
    },
-- skipped
    ["iron-beam"] = {
      type = "recipe",
      name = "iron-beam",
--      subgroup = "intermediate-product", -- get from "iron-beam" item
-- skipped
    },

Option hide_from_player_crafting control how the recipe is displayed (or not) in the player's inventory, but is not exported to a json file. I think it's worth adding this option to the json file and ignoring such a recipe when parsing. When completely ignore such recipes, then the ingredients (icons, names) from that recipes can be excluded from the general list of items.
I should probably try to fix the --factorio_lab_hacks option to disable the export of recipes with zero product and recipes exclusively for machines.

 __ITEM__kr-medium-container__
oh, loaded an old version of the page. I will try to update soon.

Now recipes with hide_from_player_crafting=true is skipped with --factorio_lab_hacks option. Example: https://hhrhhr.bitbucket.io/ (0.18.36 with Krastorio2 and 0.18.36 with Krastorio2 (--factorio_lab_hacks)).

Help me out here, it's still missing names for energy-shield-mk3-equipment and energy-shield-mk4-equipment, but as best I can tell, there are no localized names for those items inside the mod? At least, I can't find the names? I can fill them in manually but it seems odd to me.

Yes, there is some weirdness with this equipment. Localised strings from game:
energy-shield-equipment=Shield generator
energy-shield-mk2-equipment=Shield generator MK2
energy-shield-mk3-equipment=Shield generator MK3
energy-shield-mk4-equipment=Shield generator MK4

But I get it:
energy-shield-equipment=Energy shield
energy-shield-mk2-equipment=Energy shield MK2
energy-shield-mk3-equipment=__ITEM__shield-generator-mk3__
energy-shield-mk4-equipment=__ITEM__shield-generator-mk4__

As I understand it, this is due to the fact that in the mod localized strings are described in the data.raw ["energy-shield-equipment"] section, and not in data.raw.recipe or data.raw.item, where the script looks for such strings.
I will think about how to determine the correct sequence for extracting localized strings.

Thanks. It seems like it may have something to do with possibly changing the ids of those items as well, there are localized strings for shield-generator-mk1 through shield-generator-mk4, and I think that's essentially what the mod is wanting to call them, "Shield Generator" instead of "Energy Shield." I can't tell how it pulls that off but I think for now I'm manually just going to call them "Shield Generator MK1" thru "Shield Generator MK4".

@hhrhhr are you able to build for Bob's & Angel's? I'm running into problems left and right. I had to copy mod-settings.dat, then ran into a problem in angelsaddons-cab_0.2.3\prototypes\entities\equipment.lua, then logs started to flood with errors after I hacked past that problem.

angelsaddons-cab_0.2.3 only — no problem:
[INFO] mods load order received:
        core <- base <- angelsaddons-cab
[INFO] load order applied
[INFO] core and data load...
[NOTE]  loaded core.dataloader
[NOTE]  loaded core.data
[NOTE]  loaded base.data
[NOTE]  loaded angelsaddons-cab.data
[NOTE]  loaded base.data-updates
[NOTE]  loaded angelsaddons-cab.data-updates
[INFO] exporting data.raw...
[INFO] process all items... found 164 with type="item", 83 with other type
[INFO] process technologies... enabled 185 recipes
[INFO] process recipes... found 209 normal, 18 expensive, 9 hidden
[INFO] sort recipes...
[INFO]  found 91 ingridients and 199 products
[INFO] clear items...
[NOTE]  23 ingridients and 15 products left
...```

Without a specific error log, it is difficult to identify the cause. There are no errors when using Lua with patches from Factorio developers. But the usual Lua 5.3 gives me this error:
`..\dist\bin\lua53.exe: ...angelsaddons-cab_0.2.3/prototypes/entities/equipment.lua:31: bad argument #2 to 'format' (number has no integer representation)`

I suppose this is because in Lua 5.3 by default all numbers are of type double, and the mod author is trying to format them as integers.

Sorry, that's fair. The error you ran into I worked around by setting roundPresicion to 0 if it's nil before using it. The log I get is:
C:\Git\factorio-lab-tools\lua_lab>..\dist53\bin\lua53 factorio_data_dump.lua --gamedir "%factorio%" --iconsize 64 --clear --moddir mods export
[INFO] mods settins loaded
[INFO] mods load order received
[INFO] load order applied
[INFO] core and data load...
[LOG]   reset
[LOG]   stack traceback:
        mods/boblibrary_0.18.10/recipe-functions.lua:94: in field 'replace_ingredient'
        ...lates_0.18.9/prototypes/recipe/entity-recipe-updates.lua:19: in local 'func'
        factorio_data_dump.lua:251: in global 'require'
        mods/bobplates_0.18.9/data-updates.lua:1: in local 'func'
        factorio_data_dump.lua:251: in local '_require'
        factorio_data_dump.lua:528: in main chunk
        [C]: in ?
[LOG]   Recipe chemical-furnace does not exist.
[LOG]   stack traceback:
        mods/boblibrary_0.18.10/recipe-functions.lua:94: in field 'replace_ingredient'
        ...lates_0.18.9/prototypes/recipe/entity-recipe-updates.lua:35: in local 'func'
        factorio_data_dump.lua:251: in global 'require'
        mods/bobplates_0.18.9/data-updates.lua:1: in local 'func'
        factorio_data_dump.lua:251: in local '_require'
        factorio_data_dump.lua:528: in main chunk
        [C]: in ?
[LOG]   Recipe chemical-steel-furnace-from-steel-furnace does not exist.
[LOG]   stack traceback:
        mods/boblibrary_0.18.10/recipe-functions.lua:326: in field 'remove_result'
        ...lates_0.18.9/prototypes/recipe/entity-recipe-updates.lua:38: in local 'func'
        factorio_data_dump.lua:251: in global 'require'
        mods/bobplates_0.18.9/data-updates.lua:1: in local 'func'
        factorio_data_dump.lua:251: in local '_require'
        factorio_data_dump.lua:528: in main chunk
        [C]: in ?
[LOG]   Recipe steel-furnace-from-chemical-steel-furnace does not exist.
[LOG]   stack traceback:
        mods/boblibrary_0.18.10/recipe-functions.lua:265: in field 'add_result'
        ...lates_0.18.9/prototypes/recipe/entity-recipe-updates.lua:39: in local 'func'
        factorio_data_dump.lua:251: in global 'require'
        mods/bobplates_0.18.9/data-updates.lua:1: in local 'func'
        factorio_data_dump.lua:251: in local '_require'
        factorio_data_dump.lua:528: in main chunk
        [C]: in ?
[LOG]   Recipe steel-furnace-from-chemical-steel-furnace does not exist.
[LOG]   Running technology prerequisite cleanup...
[LOG]   Recipe 'fermentation-fruit' does not have a crafting_machine_tint set!
[LOG]   Recipe 'air-separation' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-glycerol' does not have a crafting_machine_tint set!
[LOG]   Recipe 'carbon-from-charcoal' does not have a crafting_machine_tint set!
[LOG]   Recipe 'petroleum-jelly' does not have a crafting_machine_tint set!
[LOG]   Recipe 'pellet-platinum-smelting' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-dinitrogen-tetroxide' does not have a crafting_machine_tint set!
[LOG]   Recipe 'geode-red-liquify' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-hydrazine' does not have a crafting_machine_tint set!
[LOG]   Recipe 'tungsten-ore-smelting' does not have a crafting_machine_tint set!
[LOG]   Recipe 'solid-sulfur' does not have a crafting_machine_tint set!
[LOG]   Recipe 'paste-cellulose' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-toluene-from-benzene' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-acetate-mixture' does not have a crafting_machine_tint set!
[LOG]   Recipe 'solid-sodium-hypochlorite' does not have a crafting_machine_tint set!
[LOG]   Recipe 'condensates-refining' does not have a crafting_machine_tint set!
[LOG]   Recipe 'algae-brown-burning-wash' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-aqueous-sodium-hydroxide' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-nitrogen-dioxide' does not have a crafting_machine_tint set!
[LOG]   Recipe 'bob-fertiliser' does not have a crafting_machine_tint set!
[LOG]   Recipe 'solid-nitroglycerin' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-epichlorhydrin' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-sulfur-dioxide-calcium-sulfate' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-nitrogen-monoxide' does not have a crafting_machine_tint set!
[LOG]   Recipe 'angels-electrode-used' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-synthesis-methanol' does not have a crafting_machine_tint set!
[LOG]   Recipe 'crystal-dust-liquify' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-perchloric-acid' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-cupric-chloride-solution' does not have a crafting_machine_tint set!
[LOG]   Recipe 'angels-water-enrichment-2' does not have a crafting_machine_tint set!
[LOG]   Recipe 'sulfuric-nitric-acid' does not have a crafting_machine_tint set!
[LOG]   Recipe 'alien-goo' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-resin-2' does not have a crafting_machine_tint set!
[LOG]   Recipe 'solid-chromate-smelting' does not have a crafting_machine_tint set!
[LOG]   Recipe 'paper-bleaching-2' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-sulfur-dioxide' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-chlorauric-acid' does not have a crafting_machine_tint set!
[LOG]   Recipe 'solid-fuel-fuel-oil' does not have a crafting_machine_tint set!
[LOG]   Recipe 'pellet-cobalt-smelting' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-dimethylamine' does not have a crafting_machine_tint set!
[LOG]   Recipe 'bio-plastic-2' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-phenol-catalyst' does not have a crafting_machine_tint set!
[LOG]   Recipe 'sodium-nitrate-acid-processing' does not have a crafting_machine_tint set!
[LOG]   Recipe 'geode-blue-liquify' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-pulping-liquor' does not have a crafting_machine_tint set!
[LOG]   Recipe 'solid-tungsten-oxide-smelting' does not have a crafting_machine_tint set!
[LOG]   Recipe 'glycerol' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-acetic-anhydride' does not have a crafting_machine_tint set!
[LOG]   Recipe 'solid-fuel-hydrazine' does not have a crafting_machine_tint set!
[LOG]   Recipe 'mineral-oil-lubricant' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-mineral-oil-catalyst' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-styrene-catalyst' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-methanol-from-wood' does not have a crafting_machine_tint set!
[LOG]   Recipe 'solid-sodium-perchlorate' does not have a crafting_machine_tint set!
[LOG]   Recipe 'hydrogen-sulfide' does not have a crafting_machine_tint set!
[LOG]   Recipe 'ethanol-to-ethylene-synthesis' does not have a crafting_machine_tint set!
[LOG]   Recipe 'kraft-recovery' does not have a crafting_machine_tint set!
[LOG]   Recipe 'coolant' does not have a crafting_machine_tint set!
[LOG]   Recipe 'silver-nitrate' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-nitric-acid' does not have a crafting_machine_tint set!
[LOG]   Recipe 'sulfite-pulping' does not have a crafting_machine_tint set!
[LOG]   Recipe 'geode-purple-liquify' does not have a crafting_machine_tint set!
[LOG]   Recipe 'solid-ammonium-perchlorate' does not have a crafting_machine_tint set!
[LOG]   Recipe 'stone-crushed-dissolution' does not have a crafting_machine_tint set!
[LOG]   Recipe 'crystal-powder-slurry' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-acid-catalyst' does not have a crafting_machine_tint set!
[LOG]   Recipe 'kraft-cooking-washing' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-chlor-methane' does not have a crafting_machine_tint set!
[LOG]   Recipe 'silver-oxide' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-white-liquor' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-fish-atmosphere' does not have a crafting_machine_tint set!
[LOG]   Recipe 'nitroglycerin' does not have a crafting_machine_tint set!
[LOG]   Recipe 'solid-sodium-hydroxide' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-ethylbenzene-catalyst' does not have a crafting_machine_tint set!
[LOG]   Recipe 'lead-oxide-2' does not have a crafting_machine_tint set!
[LOG]   Recipe 'solid-sodium-nitrate-processing' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-propionic-acid-catalyst' does not have a crafting_machine_tint set!
[LOG]   Recipe 'angelsore8-sludge' does not have a crafting_machine_tint set!
[LOG]   Recipe 'anaerobic-fermentation' does not have a crafting_machine_tint set!
[LOG]   Recipe 'solid-ammonium-nitrate' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-sulfuric-acid' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-synthesis-methanation' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-plastic-1' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-toluene-from-naphtha' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-synthesis-reforming' does not have a crafting_machine_tint set!
[LOG]   Recipe 'kraft-causting' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-polyehtylene-catalyst' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-methanol-catalyst' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-allylchlorid' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-hydrochloric-acid' does not have a crafting_machine_tint set!
[LOG]   Recipe 'processed-silver-smelting' does not have a crafting_machine_tint set!
[LOG]   Recipe 'slag-processing-dissolution' does not have a crafting_machine_tint set!
[LOG]   Recipe 'solid-sodium-cyanide' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-urea' does not have a crafting_machine_tint set!
[LOG]   Recipe 'paper-bleaching-3' does not have a crafting_machine_tint set!
[LOG]   Recipe 'bio-rubber' does not have a crafting_machine_tint set!
[LOG]   Recipe 'solid-sodium-chlorate' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-hydrogen-fluoride' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-resin-3' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-hydrofluoric-acid' does not have a crafting_machine_tint set!
[LOG]   Recipe 'solid-sodium-hydroxide-solid-sodium-sulfate' does not have a crafting_machine_tint set!
[LOG]   Recipe 'geode-yellow-liquify' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-phosgene' does not have a crafting_machine_tint set!
[LOG]   Recipe 'bio-resin-resin-liquification' does not have a crafting_machine_tint set!
[LOG]   Recipe 'solid-fuel-methane' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-benzene-catalyst' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-hydrogen-chloride' does not have a crafting_machine_tint set!
[LOG]   Recipe 'geode-lightgreen-liquify' does not have a crafting_machine_tint set!
[LOG]   Recipe 'filter-lime-used' does not have a crafting_machine_tint set!
[LOG]   Recipe 'angels-atomic-bomb-2' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-aqueous-sodium-hydroxide-reverse' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-plastic-3' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-hydrochloric-acid-solid-sodium-sulfate' does not have a crafting_machine_tint set!
[LOG]   Recipe 'angelsore9-sludge' does not have a crafting_machine_tint set!
[LOG]   Recipe 'angels-water-enrichment-1' does not have a crafting_machine_tint set!
[LOG]   Recipe 'geode-cyan-liquify' does not have a crafting_machine_tint set!
[LOG]   Recipe 'solid-trinitrotoluene' does not have a crafting_machine_tint set!
[LOG]   Recipe 'rocket-oxidizer-capsule' does not have a crafting_machine_tint set!
[LOG]   Recipe 'angels-water-enrichment-3' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-puffer-atmosphere' does not have a crafting_machine_tint set!
[LOG]   Recipe 'solid-sodium-carbonate' does not have a crafting_machine_tint set!
[LOG]   Recipe 'bio-plastic-1' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-polluted-fish-atmosphere' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-cellulose-acetate' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-acetic-acid-catalyst' does not have a crafting_machine_tint set!
[LOG]   Recipe 'aerobic-fermentation' does not have a crafting_machine_tint set!
[LOG]   Recipe 'fermentation-corn' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-melamine' does not have a crafting_machine_tint set!
[LOG]   Recipe 'solid-fuel-naphtha' does not have a crafting_machine_tint set!
[LOG]   Recipe 'solid-calcium-carbonate' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-synthesis-separation' does not have a crafting_machine_tint set!
[LOG]   Recipe 'bio-plastic' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-resin-1' does not have a crafting_machine_tint set!
[LOG]   Recipe 'solid-fuel-synthesis' does not have a crafting_machine_tint set!
[LOG]   Recipe 'brown-liquor-recovery' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-ammonia' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-methylamine' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-ammonia-from-blue-fiber' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-ferric-chloride-solution' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-bisphenol-a' does not have a crafting_machine_tint set!
[LOG]   Recipe 'cumene-process' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-hydrogen-fluoride' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-rubber-1' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-monochloramine' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-dimethylhydrazine' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-hydrogen-chloride-separation' does not have a crafting_machine_tint set!
[LOG]   Recipe 'lead-oxide' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-ammonium-chloride' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-hexafluorosilicic-acid' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-plastic-2' does not have a crafting_machine_tint set!
[LOG]   Recipe 'bio-resin' does not have a crafting_machine_tint set!
[LOG]   Recipe 'water-mineralized' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-carbon-dioxide-from-wood' does not have a crafting_machine_tint set!
[LOG]   Recipe 'liquid-naphtha-catalyst' does not have a crafting_machine_tint set!
[LOG]   Recipe 'gas-formaldehyde-catalyst' does not have a crafting_machine_tint set!
[LOG]   Recipe 'water-viscous-mud' does not have a crafting_machine_tint set!
[LOG]   Recipe 'rocket-fuel-capsule' does not have a crafting_machine_tint set!
[INFO] exporting data.raw...
[INFO] process all items... found 1588 with type="item", 262 with other type
[INFO] process technologies... enabled 2344 recipes
[INFO] process recipes...[INFO] missed loc string: ...angels-plate-glass-3 (called from L():84)
[INFO] missed loc string: ...angels-plate-glass-3 (called from L():84)
[INFO] missed loc string: ...nil (called from gsub():-1)
..\dist53\bin\lua53: factorio_data_dump.lua:625: .\factorio\locale.lua:67: attempt to concatenate a nil value (local 'id')
stack traceback:
        [C]: in function 'error'
        factorio_data_dump.lua:625: in main chunk
        [C]: in ?

Try to use -v flag and show mod load order like this:
[INFO] mods load order received:
        core <- base <- mod1 <- mod2 <- ...

Hey @hhrhhr, I've actually been able to make a lot of progress with Bob's and Angel's, at this point it seems like the only issue is that some of the icons don't get resized the way they should, which is odd to me. You can see the current sprite sheet here.
It looks like some of the Angel's sprites are messed up in row 43. I checked the output of your tool and it didn't list any problems in that row, although for a lot of the icons I get different messages like:

libpng warning: iCCP: known incorrect sRGB profile
libpng warning: Interlace handling should be turned on when using png_read_image
libpng warning: iCCP: cHRM chunk does not match sRGB

I know some of these don't matter but some seem new to me. I did use the -v flag with this run.
I did have to make some modifications to gd_image.lua to avoid errors.
local function apply_tint(ico, tint)
    local X, Y = ico:sizeXY() -- gd func
    for y = 0, X-1 do
        for x = 0, Y-1 do
            local c = ico:getPixel(x, y)
            if tint.r ~= nil and tint.g ~= nil and tint.b ~= nil then
                local r = math.floor(ico:red(c)   * tint.r)
                local g = math.floor(ico:green(c) * tint.g)
                local b = math.floor(ico:blue(c)  * tint.b)
    --            local a = math.floor(ico:alpha(c) * tint.a)
                local a = ico:alpha(c)
                c = ico:colorAllocateAlpha(r, g, b, a)
                if not c then
                    local orig = ico:getPixel(x, y)
                    if orig ~= nil then
                        c = orig
                    end
                end
            end
            ico:setPixel(x, y, c)
        end
    end
end

Sadly it takes like 30 minutes to generate this sprite sheet also :(

I have not found a way to disable such warnings other than modifying the libpng sources (this is wrong way). If there are many such warnings, then it is best to warn the author of the modification about using outdated tools. Or you can just redirect the stderr output to a file and hide it )
Regarding improperly scalable icons - what mod are they from?
I can also see that the overlay icons at this scale are rendered too small too. Probably, there is an error somewhere due to 128px icons from the mod, while all calculations are based on the fact that icons can be at most 64px. So I'm waiting for the name of the mod )

It seems to be some of angel's, angel's refining is one with a problem and i can duplicate the issue with just that mod. I think you're on the right track suspecting the 128px but haven't had enough time to troubleshoot.


PARAM_SET in SITL take a second on Mac OS

This points to something being off with locking or priorities (and not just on Mac OS, but it is just showing there).

@bkueng @julianoes I have a hunch that during SITL boot the fact that lockstep is active does interfere with app boot up until the system is fully running.

I can't reproduce this on my Mac. Which SITL vehicle are you using? How are you testing or measuring this? Which version?

Closing as there was response and I haven't seen this myself.


No way to bring window back if I close it using red dot

No way to bring window back if I close it using red dot
I have to quit and relaunch the app.
Suggestions
Either, close app on red dot
Or, add menu item to get window back and open window on app focus if it is not open

AppDelegate.swift:
func applicationShouldTerminateAfterLastWindowClosed(_ sender: NSApplication) -> Bool {
    return true
}

or similar


explicitly disallow multitaper in presence of bad annotations

closes #12519
For now, with the 1.7 release pending, @larsoner and I agreed that disallowing multitaper on Raws when there are bad_* annotations present (and reject_by_annotation=True) is the best way to go.
In the long term, we could make that combination actually work by adding a new param (chunk_size or so, analogous to n_per_seg in Welch) and basically doing a welch-MT combination (using multitaper for each chunk, then combining the estimates from each chunk at the end).  It's a bit complicated because (we think) you would need to aggregate across-chunks-within-taper first, and then agg across tapers afterward. (cc @mmagnuski who I think has done that for his research before / maybe even implemented in borsar?)

Agreed that's a good way to proceed. This doesn't resolve #11413 though, right?

Agreed that's a good way to proceed. This doesn't resolve #11413 though, right?

correct, this does not address #11413


handle non-existing firestore documents

This removes the basically unused useValidateCommentableDocument hook. It was called, which did cause it to be warmed up. But the code in the onError was not being called so the document was not being created here.
This also removes an unused useDocumentHistory hook. The document history handling was moved into the model (tree-manager.ts) after this initial hook was added a while ago.

@tejal-shah this is a framework change, there are some notes in the story: https://www.pivotaltracker.com/story/show/186498686 about how it is hard to test.
I'm going to go ahead and merge it, so there more time between now and the next release to find problems.


[WIP] Add landing pages

Related #324.

📷 Landing Page (IPFS client already running)

📷 Landing Page (IPFS Client missing)

@lidel where do you think that is the better place to store the landing pages?
In the original issue #324 you mentioned storing it at the same level as add-on/ or tests/.
But this way is necessary to slightly modify the build commands/scripts.
I was thinking about add-on/src/landing-pages/ as a better solution.

@nunofmn  that is the question:
do we ship landing pages with browser extension?
or do we publish them to IPFS and what browser extension does is to just open IPFS resource at a public gateway?
Personally I was thinking about following an example of WebUI, which is not shipped with go-ipfs but loaded from IPFS instead (go-ipfs has IPFS  path of WebUI hardcoded in sources).  I am aware that there may be use cases to make it work without internet connectivity (as suggested in https://github.com/ipfs/ipfs-companion/issues/324#issuecomment-348813189), but Landing Page will be displayed right after install/update, which never happen without working internet connection. So for this specific use case, it is safe to assume connectivity will always be present.
An argument to ship landing page with extension is a risk of public gateway (`ipfs.io) being blocked in some jurisdictions. But then, how user  is supposed to install go-ipfs/Station, if links from Landing Page point at a blocked domain?  We would have to address it somehow (eg. keep installation packages at different tld than public gw) .
I imagine there are other arguments which I missed, but for now, I think we can assume it will be shipped as webui-like IPFS resource (which means we keep it outside of  add-on/ dir).
Just to keep things decoupled and easier to maintain.
Does this sound ok?

I'm all for this! Seems like we could be doing a lot here to help new users get started and excited about distributing their webs.
I'm mildly in favour of hosting the pages on ipfs, though i don't feel too strongly about it either way.
I have more feels on the design front tho! How about something a little more like the ipfs.io site?

Once the pre-built installers for station land, i'd re-rig it to suggest installing that as per your PR. I think it's helpful to focus the user on just one next step rather than suggesting multiple options at this point in their adventure.

A bit more context... I feel like I've walked in on this PR waaay late to be all "WHAT ABOUT DIFFERENT!" :-)
I'm working on the UI for ipfs-companion more generally,( see: uploads and menu ) and I'd like to build a style guide for it so we don't have to think too hard about how new features should look.
Would you be interested up for discussing such a thing @nunofmn ?

....aaaaaaand I was thinking we could use the details section of the "pls install ipfs" page as a place we could explain that we need browsers to open up more apis to make the distributed web easier to use.

@lidel sorry for the late reply.
I still feel that shipping the landing pages in the extension is the better option, mainly from a deployment and distribution perspective.
I understand the approach used in go-ipfs, but in that case you already have the IPFS client and you can load the WebUI. In this case you may not have the IPFS client/Station installed and you cannot access "ipfs.io" gateway and you will not show the user any page when the extension is installed.
At least if the landing pages are bundled with the extension, you are able to show the user a page showing the the extension was successfully installed, even if he's not able to download IPFS Station. This way the user could try to find an alternative source to download IPFS Station, instead of simply dismiss or uninstall the extension because it didn't receive any further instructions on how to use or properly configure it.
Of course this all depends on the user that this extension is trying to target right now.
If it is targeting a developer userbase that have a proper knowledge about IPFS and the distributed web, maybe this instructions are not necessary and they will find a way of properly configuring the system.
But if we want to reach a tech-savvy userbase without a priori knowledge about IPFS and the distributed web, maybe bundling the landing pages with the extension could lead to a smoother on boarding user experience.
The update/changelog pages since are not so essential, could be loaded from IPFS directly.

@olizilla yeah I think we can workout a style guide for the extension 👍
I also like the page layout you presented but I have some observations:

I think this landing pages will be released after the IPFS Stations packages are released (@lidel please correct me if I'm wrong). So I already built the page taking into consideration that the IPFS Station installers will already be available, which will bring a smoother user on boarding experience.

In your details section maybe we could try to mimic the layout and content that I used in the Learn more and Applications sections, while also adding the browser APIs and distributed web topics you mentioned.

But lets build a style guide with a common stylesheet that could be used across the extension. 👍

@nunofmn I agree, this is a valid and quite important concern:

if the landing pages are bundled with the extension, you are able to show the user a page showing the the extension was successfully installed, even if he's not able to download IPFS Station through the provided link. This way the user could try to find an alternative source to download IPFS Station, instead of simply dismiss or uninstall the extension because it didn't receive any further instructions on how to use or properly configure it.

I am guilty of forgetting that we already had an example of a government blocking public IPFS gateway.  To work around such events in future we actually have no choice, but to ship landing page with exception itself.
Thank you for pointing this out.

As for use cases around landing page, yes, we will ship it when Station has  one-click install packages for all platforms.  So we want to design the message as if it was already ready for use.

@nunofmn @lidel good points. I agree. Shipping these pages with the addon makes more sense.
As for a style guide, we've been using tachyons as the starting point. It's an implementation of atomic css which is fun, but the killer feature is the amount of research that has gone in to the default values for all of the building blocks. I'm working on an ipfs flavoured theme on top of it.

@olizilla nice. So maybe I will restructure the landing pages to use tachyons, and then when you finish the IPFS theme based on tachyons we uniformize the style of the extension. I could help in
@lidel since we are shipping the landing pages with the extension, maybe we should move the landing pages folder to add-on/src/?

Agreed, it makes sense now to move them.
I assume we are going to use/share browserified assets such as tachyons,  so does something like add-on/src/pages/ sound good?

@lidel do you think that we should stick to static pages? Or maybe build the pages using choo?

@nunofmn we will have some context-dependent content, namely a section informing user that Station/go-ipfs should be installed – it will be visible only  if browser extension is unable to connect to API.
This is simple enough that could be handled by a static page, but we may want to anticipate future needs for other use cases, such as "displaying some engaging stats about running node".
It may be more future-proof to go with choo (if that is not a big chore), but I don't have a strong opinion yet.
If you choose to use choo, event-based code would be similar to our browser action menu, which shows/hides menu items depending on current tab being IPFS path or not, displays peer count/offline status depending on API connectivity etc.

📷 Landing Page using Tachyons (IPFS client already running)

@lidel I also think that building the static pages with choo is a more future-proof solution. When I finish the layout the I will migrate it to choo 👍

📷 Landing Page using Tachyons (IPFS client missing)

Just fyi we started shipping parts of UI Kit in add-on/ui-kit
(Not in git, directory is created from external deps during npm run build:copy:ui-kit):
add-on/ui-kit
├── fonts
│   └── *.woff2
├── ipfs.css
└── tachyons.css

Related Resources

Designing Styleguide and Language for the UI's: https://github.com/ipfs-shipyard/pm-ipfs-gui/issues/7

You can also see it and CSS inspect on Zeplin: https://zpl.io/awyMrgj
(comment in issue above if you dont have access)

WIP  implementation of the above (UI Kit): https://github.com/ipfs-shipyard/ipfs-css

Closing this, as it got superseded by https://github.com/ipfs-shipyard/ipfs-companion/pull/565 which just landed :)
Big thank you to everyone contributing ideas, pocs and  mockups to both PRs!


Synced with upstream ipfs-service-provider

:tada: This PR is included in version 1.11.0 :tada:
Your semantic-release bot :package::rocket:


Adds functions to track nodes' secondary machine accounts

The FlowEpoch contract now tracks where collector and consensus nodes' qc and dkg objects are stored in the contract.
This dictionary is stored in the contract storage in order to make loading the contract object more efficient.
I also picked up the latest changes from master in this PR

closing this to make the changes to the staking collection instead


想要新数据

可以给我一些其他数据吗，我想测试以下代码的性能

Can you give me some additional data, I want to test the performance of the following code


Favicon.ico reference not generated in pages when site builds

I remember favicon.ico showing up sometime - not sure how to get that from the toml down into the rendered pages

I added a note in the (default blog starter](https://github.com/WaylonWalker/markata-blog-starter/blob/main/pages/your-own-style.md), which is probably the best starting guide I have.
Also added it to the theming page of the docs, both seem not the most discoverable.


add dependency in build.gradle (app)

compile 'com.android.support:design:23.3.0'
if not having this, in the MainActivity.java will report 'can't resolve android.support.design.widget'

Hi, the original build.gradle file has "compile 'com.android.support:design:23.3.0'" please check the latest code of the project. Thanks!

Thanks for your attention. Maybe some information is missed during creating the pull request. Actually I mean the gradle script in readme.md, not the build.gradle in the project. In fact, I find out this issue just through comparing the build.gradle in the project and the gradle script in readme.md.

Thanks for your advice, I already update the build.gradle setting in readme.md file to be consistant with the project. Please check, thanks!

Since the problem is solved, close the issue now.


Installing mvIMPACT

Hello fellow Harvesters!
I'm trying to follow along with the installation process for using Harvesters and am embarrassingly stuck at the the mvIMPACT step. I've downloaded the mvIMPACT_AcquireMergeModules zip folder, but I'm not sure where to extract them to. Should I place them in my Anaconda environment dedicated to using Harvesters?
I apologize for the simple question, but any help would be greatly appreciated.

@jmdelahanty Hi, I do not know which OS you are running but you need to grab mvGenTL_Acquire-.. These packages have an installer or an installer script. Once the package is installed there you will find a CTI file and that is the file you need to load on the Harvester object. Keep in mind that a CTI file is a DLL / a shared object so please leave them where they were. If you move them then they would fail to work. Good luck! Regards, Kazunari.

Excellent timing as I too just got stuck on this point of install.  The instructions are clearly calling for mvIMPACT... but should be changed to mvGenTL...   I was guessing this would be the needed file since it's the only one with the installer extension (msi).
Related to this step, the instructions specify version 2.29.0 and I inadvertently installed 2.29.1 and I would hope the difference is minor.  However, the latest version is at 2.42.0 so would you recommend sticking with 2.29.x?  Or will it likely be safer/better to use the latest version?

@barriebarry Hi, they have just changed the package name for some business reasons. I will revise the information but please note that it is not possible to make everything up to date because I’m running this project on a hobby basis. Concerning the version number, I’d not mind go with the latest version as long as I know who MATRIX VISION are. Regards, Kazunari.

Thanks for the comments barrie and Kazunari! I think I've installed mvGenTL successfully using their installer. I'm on Windows10 by the way. I'm getting stuck on the iPython example now.
When I do the following...
Python 3.8.8 (default, Feb 24 2021, 15:54:32) [MSC v.1928 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.22.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from harvesters.core import Harvester

In [2]: import numpy as np

In [3]: h = Harvester()

In [4]: h.add_file("C:/Users/jdelahanty/.conda/envs/genicam/Lib/site-packages/genicam/TLSimu.cti")

In [5]: h.update()

In [6]: h
Out[6]: <harvesters.core.Harvester at 0x1695cf83520>

In [7]: len(h.device_info_list)
Out[7]: 0

it looks like I'm not getting any devices added to the system. Any advice?

@jmdelahanty I'm at the same point of getting zero in response to len(h.device_info_list).  And I see you just discovered what I was about to point out that you need to down grade to Python 3.7 yet doing that did not solve the problem.  Here are instructions on how to change the version of Python using Anaconda:
https://stackoverflow.com/questions/43592879/how-to-change-python-version-in-anaconda-spyder
Also, this is sort of described in issue #125 in the closed tickets which would be useful to have this ticket pegged at the top.

Interesting that this also didn't solve the problem.
I'm currently trying to find the .cti file from the mvGenTL install to use the camera directly since I can't get TLSimu to work yet.
Unfortunately, there doesn't appear to be one from the install. Perhaps I've installed the incorrect thing accidentally?
I installed it for all users on this machine so it's been placed in a Public folder in this path:
C:\Users\Public\Documents\MATRIX VISION\mvIMPACT Acquire\ GenICam
And I get the following files:
MATRIXVISION_GenTL_Interface_12_0_1 and MATRIXVISION_GenTL_System_9_0_0
There's also a Cache folder but all that's present is two bin files that I don't know what they are for/do. Is that something you're seeing as well @barriebarry ?

If you are trying to get it working with a U3V camera then check if you have installed the U3V driver to the target camera. This procedure will bridges between Harvester and MV’s GenTL Producer and the target U3V camera. I don’t mind if you play with TLSimu but that’s for my testing purpose. Our Wiki or Issue page should have mentioned your missing piece but I do not think there would learn much from TLSimu so I would recommend to work with your real target.

Okay, got it. I'll continue to try doing that.
I'm using a Teledyne DALSA Genie Nano and it's connected through ethernet using a power-over-ethernet protocol. I'm not sure if that makes it a U3V device or not, I'm pretty new to this.
The mvGigE Configure program recognizes the camera, but I just can't seem to find the .cti file. Do you think I [downloaded/installed] the wrong thing Kazunari?
I'd attach a screenshot of the camera being connected, but it doesn't look like I can attach photos here...

Your target is GEV camera. Okay, so have made sure you can acquire images from DALSA’s app?

Okay! That's great to know, thanks for teaching me.
I have! I get images through the Sapera CamExpert app. I've also successfully gotten images using MATLAB, but am hoping to use Python to control the camera alongside our work.

Where did you search to find out the CTI file? I guess you’ll find it in their installation folder. Of course, I’m not fully sure because I’ve not been working with their package as I didn’t know the name change.

In addition, there still be a case where they have hidden the CTI file for some business reasons.
Anyway, just execute a search in the installation folder, and eventually in the whole hard disk if you can’t find it in the installation folder.

I was looking in the wrong folder! I got the iPython example to work by looking in the correct location noted below:

Thank you Kazunari!

Thanks for the help and sharing the details.  I managed to make it work too.


Hieght of Symbol

There is a feeling that auto-fit only works on the number of characters. But if 1 big symbol with the large height then auto-fit not working.

For example
<me.grantland.widget.AutofitTextView             android:layout_width="200dp"             android:layout_height="20dp"             android:maxLines="1"<EMAIL_ADDRESS>           android:text="5"/>

+1


gui: typo fix in README

You need fix the DCO.  Please see the details link for that.  In the future 'git commit -s' avoids this issue.

@maliberty ah sorry about that, thanks!

Thanks


Target is not part of cloudflare network

Hi!
First thanks for CF, definitely a dope tool.
Do you know why would CF say a website is not part of Cloudflare network but I am seeing an error 1016 page from CloudFlare of said target?
Thanks a lot beforehand.

Hi there,
Thank you for your kind words!  Sorry I did not respond sooner, do you still need assistance with this?

I do, yes. I got the same problem.


No type validation of cluster-opt in swarm manage and no output when incorrect type

Hi, all
I found that when swarm manage works, it will not validate the correct type of cluster options cluster-opt.
For example, when we set swarm.createretry=-1, -1 is an int and swarm will check if this value is larger than 0.
While, when we set swarm.createretry=ssss, ssss is not an int and swarm will check this options but nothing will output to let administrator know it is in incorrect format.

DEBU[0000] Registering HTTP route                        method=DELETE route=/images/{name:.*}
DEBU[0000] Registering HTTP route                        method=DELETE route=/networks/{networkid:.*}
DEBU[0000] Registering HTTP route                        method=DELETE route=/volumes/{name:.*}
DEBU[0000] Registering HTTP route                        method=HEAD route=/containers/{name:.*}/archive
DEBU[0000] Registering HTTP route                        method=GET route=/containers/{name:.*}/stats
DEBU[0000] Registering HTTP route                        method=GET route=/volumes
DEBU[0000] Registering HTTP route                        method=GET route=/containers/{name:.*}/changes
DEBU[0000] Registering HTTP route                        method=GET route=/containers/{name:.*}/logs
DEBU[0000] Registering HTTP route                        method=GET route=/containers/{name:.*}/json
DEBU[0000] Registering HTTP route                        method=GET route=/volumes/{volumename:.*}
DEBU[0000] Registering HTTP route                        method=GET route=/events
DEBU[0000] Registering HTTP route                        method=GET route=/containers/{name:.*}/export
DEBU[0000] Registering HTTP route                        method=GET route=/images/viz
DEBU[0000] Registering HTTP route                        method=GET route=/images/get
DEBU[0000] Registering HTTP route                        method=GET route=/images/{name:.*}/get
DEBU[0000] Registering HTTP route                        method=GET route=/images/{name:.*}/history
DEBU[0000] Registering HTTP route                        method=GET route=/containers/json
DEBU[0000] Registering HTTP route                        method=GET route=/containers/{name:.*}/archive
DEBU[0000] Registering HTTP route                        method=GET route=/_ping
DEBU[0000] Registering HTTP route                        method=GET route=/version
DEBU[0000] Registering HTTP route                        method=GET route=/networks
DEBU[0000] Registering HTTP route                        method=GET route=/images/search
DEBU[0000] Registering HTTP route                        method=GET route=/images/{name:.*}/json
DEBU[0000] Registering HTTP route                        method=GET route=/containers/ps
DEBU[0000] Registering HTTP route                        method=GET route=/containers/{name:.*}/top
DEBU[0000] Registering HTTP route                        method=GET route=/containers/{name:.*}/attach/ws
DEBU[0000] Registering HTTP route                        method=GET route=/exec/{execid:.*}/json
DEBU[0000] Registering HTTP route                        method=GET route=/info
DEBU[0000] Registering HTTP route                        method=GET route=/images/json
DEBU[0000] Registering HTTP route                        method=GET route=/networks/{networkid:.*}
DEBU[0000] Registering HTTP route                        method=POST route=/containers/{name:.*}/update
DEBU[0000] Registering HTTP route                        method=POST route=/containers/{name:.*}/copy
DEBU[0000] Registering HTTP route                        method=POST route=/commit
DEBU[0000] Registering HTTP route                        method=POST route=/build
DEBU[0000] Registering HTTP route                        method=POST route=/networks/{networkid:.*}/disconnect
DEBU[0000] Registering HTTP route                        method=POST route=/containers/{name:.*}/rename
DEBU[0000] Registering HTTP route                        method=POST route=/containers/{name:.*}/exec
DEBU[0000] Registering HTTP route                        method=POST route=/exec/{execid:.*}/start
DEBU[0000] Registering HTTP route                        method=POST route=/containers/{name:.*}/kill
DEBU[0000] Registering HTTP route                        method=POST route=/containers/{name:.*}/pause
DEBU[0000] Registering HTTP route                        method=POST route=/images/{name:.*}/push
DEBU[0000] Registering HTTP route                        method=POST route=/containers/{name:.*}/unpause
DEBU[0000] Registering HTTP route                        method=POST route=/containers/{name:.*}/resize
DEBU[0000] Registering HTTP route                        method=POST route=/exec/{execid:.*}/resize
DEBU[0000] Registering HTTP route                        method=POST route=/volumes/create
DEBU[0000] Registering HTTP route                        method=POST route=/images/create
DEBU[0000] Registering HTTP route                        method=POST route=/images/load
DEBU[0000] Registering HTTP route                        method=POST route=/images/{name:.*}/tag
DEBU[0000] Registering HTTP route                        method=POST route=/containers/{name:.*}/restart
DEBU[0000] Registering HTTP route                        method=POST route=/containers/{name:.*}/start
DEBU[0000] Registering HTTP route                        method=POST route=/containers/{name:.*}/wait
DEBU[0000] Registering HTTP route                        method=POST route=/networks/create
DEBU[0000] Registering HTTP route                        method=POST route=/networks/{networkid:.*}/connect
DEBU[0000] Registering HTTP route                        method=POST route=/auth
DEBU[0000] Registering HTTP route                        method=POST route=/containers/create
DEBU[0000] Registering HTTP route                        method=POST route=/containers/{name:.*}/stop
DEBU[0000] Registering HTTP route                        method=POST route=/containers/{name:.*}/attach
DEBU[0000] Registering HTTP route                        method=PUT route=/containers/{name:.*}/archive
WARN[0000] WARNING: rescheduling is currently experimental, use at your own risks
DEBU[0000] Watchdog enabled
INFO[0000] Listening for HTTP                            addr=<IP_ADDRESS>:2375 proto=tcp
DEBU[0000] Start monitoring events                       id=RDEJ:ZCV6:CJH5:UCNB:P5RR:66LB:JZSO:YACX:TBL3:PYMV:2F77:KBUA name=ubuntu
INFO[0000] Registered Engine ubuntu at <IP_ADDRESS>:2376

As we set ssss to  swarm.overcommit which is invalid, I think we should show something in log.

@allencloud Good catch!
Feel free to submit a PR to check those two options.

@jimmyxian
Actually, I am on this work.

ping @vieux @dongluochen @MHBauer @nishanttotla
I think this is an important feature swarm needs to implement. WDYT


qunit tests

Hi!
I have a qunit testsuite for a complete site. How do I (if ever possible) run it under testling.com?
TIA,
--Vladimir

Pretty soon you'll just be able to include it from npm like this: http://testling.com/docs/#using-npm
but I need to write the qunit harness module first and publish it to npm.

I don't think the harness module was ever created, and I'm not sure if qunit is still needed, so closing. Happy to reopen if this is still needed.


BBS spk security against the issuer

Closes #75
Adding a section to the security considerations to document the fact that a bbs spk is not zero knowledge against the issuer,

Multiple approvals, discussed on WG call, purely editorial non-normative content, merging


PoAMiner mines for most of turn rather than only once

Rather than nodes only mining right on their timestamp, they may have the ability to retry during their turn. They have a window of time during which to mine.

This is still relevant.
Just waiting for <IP_ADDRESS> to be all cleaned up and then will test this and the other smart contract changes inside master.

@codingupastorm Can you perhaps latest from master and give us an update on what we  should do with this PR?

Yep! It's worth putting in before any more smart contract performance testing happens

@codingupastorm awesome! I assume needs to go to 3.0.8 as well?


Add head http method.

Hello, I was looking for a drop-in replacement for httpie, which is fantastic but add a 300ms overhead to every request on my work machine, and I think ht should be ideal for helping me develop my APIs. I use HEAD endpoints in my APIs so I needed this method included. I'm not a seasoned Rust developer so it may not be perfect code.

Thanks, @plombard. It seems this PR is overlapping with another PR that is adding support for the HEAD HTTP method. Maybe you could update this PR to support the OPTIONS method instead? Thanks again.

Sure, I'll give it a try.


Add su accounts to oss

Codecov Report

Merging #219 (e2c7b45) into master (807a1dc) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #219   +/-   ##
=======================================
  Coverage   78.37%   78.37%
=======================================
  Files           1        1
  Lines          37       37
=======================================
  Hits           29       29
  Misses          4        4
  Partials        4        4

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 807a1dc...e2c7b45. Read the comment docs.


Calling get_results after insert does not compile (postgres)

Hi,
First, thank you for this crate.
I've found a case where the code should compile but does not. Of course correct me if I'm wrong.
I'm just calling get_results after the insert statement on a nullable or not enum mapping.
The issue occurs only if I use #[ExistingTypePath = "MyRemoteEnumMapping"] to map an existing enum.
I've pushed the test case in a branched fork here : https://github.com/rlamarche/diesel-derive-enum/blob/issue-insert-get-results/tests/src/pg_remote_nullable.rs
The full diagnostic from Rust is :
error[E0277]: the trait bound `(diesel::sql_types::Integer, diesel::sql_types::Nullable<pg_remote_nullable::MyRemoteEnumMapping>): load_dsl::private::CompatibleType<RemoteMaybeNullable, Pg>` is not satisfied
    --> src/pg_remote_nullable.rs:117:45
     |
117  |         .get_results::<RemoteMaybeNullable>(connection)
     |          -----------                        ^^^^^^^^^^ the trait `load_dsl::private::CompatibleType<RemoteMaybeNullable, Pg>` is not implemented for `(diesel::sql_types::Integer, diesel::sql_types::Nullable<pg_remote_nullable::MyRemoteEnumMapping>)`
     |          |
     |          required by a bound introduced by this call
     |
     = help: the following other types implement trait `load_dsl::private::CompatibleType<U, DB>`:
               (ST0,)
               (ST0, ST1)
               (ST0, ST1, ST2)
               (ST0, ST1, ST2, ST3)
               (ST0, ST1, ST2, ST3, ST4)
               (ST0, ST1, ST2, ST3, ST4, ST5)
               (ST0, ST1, ST2, ST3, ST4, ST5, ST6)
               (ST0, ST1, ST2, ST3, ST4, ST5, ST6, ST7)
             and 24 others
     = note: required for `InsertStatement<table, BatchInsert<Vec<ValuesClause<(..., ...), ...>>, ..., ..., false>>` to implement `LoadQuery<'_, _, RemoteMaybeNullable>`
     = note: the full type name has been written to '/home/romain/code/vendor/diesel-derive-enum/tests/target/debug/deps/tests-c1498017a76849a6.long-type-17151896408522956576.txt'
note: required by a bound in `get_results`
    --> /home/romain/.cargo/registry/src/index.crates.io-6f17d22bba15001f/diesel-2.1.0/src/query_dsl/mod.rs:1739:15
     |
1737 |     fn get_results<'query, U>(self, conn: &mut Conn) -> QueryResult<Vec<U>>
     |        ----------- required by a bound in this associated function
1738 |     where
1739 |         Self: LoadQuery<'query, Conn, U>,
     |               ^^^^^^^^^^^^^^^^^^^^^^^^^^ required by this bound in `RunQueryDsl::get_results`

I'll look inside the code to find if I can fix it, but if I don't post update here, it means I had'nt find how to fix it.

I've found my error, and it is not a bug, neither a problem with this crate.
When calling get_results, the return type have to map every fields in the table.
Moreover, there was an error in my test case.
Sorry for the inconvenience.