Update lua vessels capabilities

This is a WIP for having more functions available in lua through the ScriptVessel.
Should solve issues #421 and #422.

Coming to a point where it becomes harder to test things (VC stuff at the moment). Is there a toy example that I could use to port to lua somewhere?

I've been fighting with cross module calls to replicate code like this :
void Atlantis::SeparateBoosters (double met)
{
	OBJHANDLE hET = GetDockStatus (GetDockHandle (1));
	if (hET) {
		VESSEL *pET = oapiGetVesselInterface (hET);
		((Atlantis_Tank*)pET)->SeparateSRBs();
	}
}

The difficulty is that each vessel lives in its own lua interpreter so you don't have access to code/data from other vessels (even of the same class).
After several false starts, I came up with a solution which forwards calls between interpreters but the syntax is not pretty and I don't know if it is completely supported by lua. It would look like that for the example above :
function SeparateBoosters()
  hET = vi:get_dockstatus(vi:get_dockhandle())
  if hET then
    pET = vessel.get_interface(hET)
    pET:import("SeparateSRBs")()
  fi
end

It basically works via two functions. lua_import which is added to the ScriptVessel lua instance and returns a closure specific to the method/interpreter
// Called from e.g. V:import("SeparateSRBs")
// Returns a closure that will call "SeparateSRBs" inside V's ScriptVessel context
static int lua_import(lua_State* L)
{
	ScriptVessel* sh = lua_toScriptVessel(L, 1);
	if (!sh)
		return 0;

	const char *method = lua_tostring(L, 2);

	// Check if the function exists in the target lua_State
	lua_State* Ltgt = sh->GetState();
	lua_getfield(Ltgt, LUA_GLOBALSINDEX, method);
	if (lua_isfunction(Ltgt, -1) == 0) {
		lua_pop(Ltgt, 1);
		lua_pushnil(L);
		return 1;
	}
	lua_pop(Ltgt, 1);

	// push method name and target lua_State as upvalues for the closure
	lua_pushstring(L, method);
	lua_pushlightuserdata(L, Ltgt);

	// return a closure that will forward the method call to the target lua_State
	lua_pushcclosure(L, lua_crosscall, 2);

	return 1;
}

and lua_crosscall which performs the grunt work :
// Extract the arguments from the original lua state,
// push them into the target lua state and call the method
// then finally push back the result from the target state to the original state
static int lua_crosscall(lua_State* L) {
	// Recover method name and lua_State for the target interpreter
	const char* method = lua_tostring(L, lua_upvalueindex(1));
	lua_State* Ltgt = (lua_State*)lua_touserdata(L, lua_upvalueindex(2));

	// Save target interpreter stacksize (used later to compute the number of results)
	int stacksize = lua_gettop(Ltgt);

	// Push the function to be called in the target
	lua_getfield(Ltgt, LUA_GLOBALSINDEX, method);

	// Number of arguments pushed in the original lua_State
	int nargs = lua_gettop(L);

	// "Forward" arguments to the target lua_State (Lto)
	// Warning : only basic types are supported (no tables/userdata)
	for (int i = 1; i <= nargs; i++) {
		switch (lua_type(L, i)) {
		case LUA_TNIL:
			lua_pushnil(Ltgt);
			break;
		case LUA_TBOOLEAN:
...
		default:
			// Warning : we push nil for unsupported args to simplify processing (no need to clean up)
			lua_pushnil(Ltgt);
			break;
		}
	}
	// Clean up original state
	lua_pop(L, nargs);

	// Call the function in the target lua state
	lua_pcall(Ltgt, nargs, LUA_MULTRET, 0);
	int nret = lua_gettop(Ltgt) - stacksize; // number of results

	// "Forward" results to the original lua_State *in reverse order*
	for (int i = nret; i >= 1; i--) {
		switch (lua_type(Ltgt, -i)) {
		case LUA_TNIL:
			lua_pushnil(L);
			break;
...
		default:
			lua_pushnil(L);
			break;
		}

	}

	// Cleanup target state
	lua_pop(Ltgt, nret);

	return nret;
}

What do you think of this solution? It seems to work, at least with toy examples (calling a luavessel function from a LuaMFD and getting back a couple of values).
There is a limitation where only basic types are supported (don't want to bring in a complete serialization library for this) but I don't think it's that big of an issue.
My concern is if a module function called this way tries to call back into the original module, this could get ugly... Of course it needs a lot more testing (especially to see if it plays nice with coroutines) before I can even put this as a commit in the PR so I'd appreciate if you have any feedback.
Maybe I overthought this and there is a simpler solution?

I haven't done any work with the LUA before so I am unfamiliar with many of the concepts and issue, but at least @schnepe2 has participated in LUA development.

Hi,
I am trying to help where I can, but I think Lua ... it's not my strong foot.
Some thoughts however:
It looks like LuAtlantis is just a 1:1 port of Atlantis to Lua. Is this the right way? I would rather think that adding an Lua Interface to Atlantis would make more sense. Any change/impovement in Atlantis you have to do again in LuAtlantis (and vice versa). So from a "do not invent the wheel again" standpoint it seems to be less than perfect.

Gave it some more thought and ended up detecting global functions automatically so there is no need for the import("xxx") ugliness anymore

I would also recommend not exposing internal states of one Lua instnce to another, but I cannot really tell why 😉

Some thoughts however: It looks like LuAtlantis is just a 1:1 port of Atlantis to Lua. Is this the right way? I would rather think that adding an Lua Interface to Atlantis would make more sense. Any change/impovement in Atlantis you have to do again in LuAtlantis (and vice versa). So from a "do not invent the wheel again" standpoint it seems to be less than perfect.

The goal is to improve Lua support to the point where it can be a viable alternative to C++ modules. Some kind of prototype vessel is required to cover the added Lua functionalities.
I chose to port the Atlantis 1 to 1 because :

I'm not a modder and acquiring the skills to create a complex vessel from scratch would take too much time for the task at hand (not that I would mind being able to do that)
It's one of the few existing vessels under the same MIT license as the Orbiter project (obviously since it's part of it) so I can put it in the repo as an example of how to implement a relatively complex vessel in Lua
It provides a reference to check against when there are odd behaviours with the Lua version

I would also recommend not exposing internal states of one Lua instnce to another, but I cannot really tell why 😉

Don't worry, only global functions are accessible. You just have to declare private stuff as local for it to be hidden.
It's my first experience with Lua so I'm pretty much playing it by ear at the moment, don't hesitate to tell if I do something stupid^^

Saw that clbkSaveState/clbkLoadStateEx capabilities were added, but there were some test problems? Are these available in a release build, or are all of these methods going to be released together at a later time? I'm just unfamiliar with GitHub and don't know what capabilities are publicly available. Thanks for working on these Lua capabilities!

Saw that clbkSaveState/clbkLoadStateEx capabilities were added, but there were some test problems?

Just automatic CI[*] testing was not working. The functionality can only tested locally.
Nevertheless the functions are available in master as they heve been merged.

Are these available in a release build, or are all of these methods going to be released together at a later time?

Yes they are available, as long as the builds were created from master (which they are). Those builds contain all the changes that were merged/pulled into the master - so "latest release" contains all the "latest changes" (including latest bugs 😉 )

I'm just unfamiliar with GitHub and don't know what capabilities are publicly available. Thanks for working on these Lua capabilities!

No problem, the distributed development via github is another level of complexity.
[*] Continous Integration

Oh sorry, I was thinking of another PR... The clbkSaveState/clbkLoadStateEx functions have only been added to this PR.
So they are only available in builds from this "branch". As it is currently only on the creators repository it is NOT included in the master builds (yet).
The builds you see in this PR (Those of the checks) however are not stored, so you cannot download them (as far as I know).
Sorry for the confusion.

Oh sorry, I was thinking of another PR... The clbkSaveState/clbkLoadStateEx functions have only been added to this PR. So they are only available in builds from this "branch". As it is currently only on the creators repository it is NOT included in the master builds (yet).
The builds you see in this PR (Those of the checks) however are not stored, so you cannot download them (as far as I know).
Sorry for the confusion.

No worries. I've been updating my Lua vessel code in anticipation of these features. Looking forward to seeing these in the master build.

Oh sorry, I was thinking of another PR... The clbkSaveState/clbkLoadStateEx functions have only been added to this PR. So they are only available in builds from this "branch". As it is currently only on the creators repository it is NOT included in the master builds (yet).
The builds you see in this PR (Those of the checks) however are not stored, so you cannot download them (as far as I know).
Sorry for the confusion.

No worries. I've been updating my Lua vessel code in anticipation of these features. Looking forward to seeing these in the master build.

I hope I'll be able to change the PR to ready status next weekend so it can be one step closer to general availability.
Sadly if you want to play with it you must compile it from this branch.

The MFD work is pushed. Since the registered MFDs are Lua objects (tables), I could not reuse the ScriptMFD since it expects a script file and uses global functions. I had to create another "VesselMFD" container to encapsulate the C++ MFD along with the Lua class.

The initial goal is pretty much done, there is still some missing API for the 2D panel views but I think it would best be done in another PR, this one is big enough already.
I still have to complete the vessel/oapi documentation, since it's really tedious, I'd like to have some feedback before doing it. It's not something I want to do twice...

The initial goal is pretty much done, there is still some missing API for the 2D panel views but I think it would best be done in another PR, this one is big enough already. I still have to complete the vessel/oapi documentation, since it's really tedious, I'd like to have some feedback before doing it. It's not something I want to do twice...

I'm not super familiar with github or how the code is managed, just a highly interested user. But if there is something I can do to facilitate the review and documentation I'm happy to help.

The initial goal is pretty much done, there is still some missing API for the 2D panel views but I think it would best be done in another PR, this one is big enough already. I still have to complete the vessel/oapi documentation, since it's really tedious, I'd like to have some feedback before doing it. It's not something I want to do twice...

I'm not super familiar with github or how the code is managed, just a highly interested user. But if there is something I can do to facilitate the review and documentation I'm happy to help.

This is quite a paradox, the idea is to be able to work without a C++ environment yet you need to compile the branch to be able to test it...
Maybe if you can grab the latest OpenOrbiter, I can attach the DLLs (LuaInline, LuaInterpreter and ScriptVessel) ?

The initial goal is pretty much done, there is still some missing API for the 2D panel views but I think it would best be done in another PR, this one is big enough already. I still have to complete the vessel/oapi documentation, since it's really tedious, I'd like to have some feedback before doing it. It's not something I want to do twice...

I'm not super familiar with github or how the code is managed, just a highly interested user. But if there is something I can do to facilitate the review and documentation I'm happy to help.

This is quite a paradox, the idea is to be able to work without a C++ environment yet you need to compile the branch to be able to test it... Maybe if you can grab the latest OpenOrbiter, I can attach the DLLs (LuaInline, LuaInterpreter and ScriptVessel) ?

I can do that. I have the x86 602931718 (February 1) build which I believe is the latest version.
But you have done a lot of code revision - what exactly would I be reviewing/testing? What do you need me to do?

OK. I installed the DLLs and clbk_savestate works correctly for my Mirage2000 vessel.
There seems to be some problem with clbk_loadstateex. I added the function but left it empty. I assume the second argument must be the vessel instance handle (vi in my code)?
function clbk_loadstateex(scn, vi) end
On start Orbiter terminates with a critical error dialog, and I get the attached Orbiter.log.
Orbiter.log
It looks like LuaInLine is trying to read a filename out of the scenario file (???). Not sure what to make of this.
I can work on converting over the keypress functions once we get this sorted.

I am also looking at  vi:create_variabledragelement to replace the add_force calls for landing gear drag. Looking at the Atlantis code it seems that there are only two arguments (the force magnitude and the vector of the point of attack). In the C++ API reference this method has a third argument of the control parameter, which I would put in as the animation state of the gear. I'm not sure how this can be a variable component without that.

OK. I installed the DLLs and clbk_savestate works correctly for my Mirage2000 vessel.

Nice!

There seems to be some problem with clbk_loadstateex. I added the function but left it empty. I assume the second argument must be the vessel instance handle (vi in my code)? function clbk_loadstateex(scn, vi) end On start Orbiter terminates with a critical error dialog, and I get the attached Orbiter.log. Orbiter.log
It looks like LuaInLine is trying to read a filename out of the scenario file (???). Not sure what to make of this.

If you implement the clbk_loadstateex callback, then you absolutely need to forward the unrecognized lines to the default scenario handler (this is the same in C++) or the Orbiter parsing logic will not correctly detect the end of the vessel declaration and will consider the second line as a new ship declaration (hence the "No vessel class configuration file found for: STATUS Landed Earth" error).
For example :
function clbk_loadstateex(scn, vs2)
     for line in scenario_lines(scn) do
		if scenario_line_match(line, "CONFIGURATION %d", match) then
			status = match.res[1]
		elseif scenario_line_match(line, "GEAR %d %f", match) then
			gear_state.status = match.res[1] + 1
			gear_state.proc = match.res[2]
                else
                        vi:parse_scenario_line_ex(line, vs2)  -- unrecognised option - pass to Orbiter's generic parser
                end
     end
end

Also there is already a vi global variable populated by default by the ScriptVessel that corresponds to the current vessel object. I would advise not using vi as a parameter or variable in your code as it will probably lead to problems.

I am also looking at vi:create_variabledragelement to replace the add_force calls for landing gear drag. Looking at the Atlantis code it seems that there are only two arguments (the force magnitude and the vector of the point of attack). In the C++ API reference this method has a third argument of the control parameter, which I would put in as the animation state of the gear. I'm not sure how this can be a variable component without that.
EDIT: Nevermind, I see you have a separate set command to set the drag level. It appears to work correctly, but I don't know if that will be a point of confusion for developers trying to use the 3-argument form of CreateVariableDrag.

Yes, I'm not really happy with that solution but since there are no references in Lua, I seemed to me to be the best compromise. I'll stress that point in the documentation

Yes, I'm not really happy with that solution but since there are no references in Lua, it seemed to me to be the best compromise. I'll stress that point in the documentation

OK. I'm unfamiliar with the inner workings of the code and I don't understand what you mean by "no references" being a problem. Is that something you can explain in plain language? Just curious.

Edit: if you want to use the scenario_lines/scenario_line_match functions, you need to put this oapi_init.lua file in your Script directory

OK. But I should still be able to parse the scn file with just Lua syntax, yes?

Edit2 : I missread your comment about the second argument. No, it's not the vessel interface, it's an opaque handle to a VESSELSTATUS2 structure that you need to provide to the parse_scenario_line_ex method.

OK. Again, the terminology is over my head. Is the handle vs already available or does it need to be referenced/defined in some way?

OK. I installed the DLLs and clbk_savestate works correctly for my Mirage2000 vessel.

Nice!

Actually, it mostly works in that it can write out numerical animation statuses and states, but it isn't writing out strings:
function clbk_savestate(scn) . . . oapi.writescenario_string(scn, "afterburner", afterburner) -- this is ignored for some reason? afterburner boolean does exist . . . end

Actually, it mostly works in that it can write out numerical animation statuses and states, but it isn't writing out strings:
oapi.writescenario_string(scn, "afterburner", afterburner) -- is ignored

Are you sure your afterburner variable is a string and not a boolean? If not you must convert it with tostring(afterburner)

OK. But I should still be able to parse the scn file with just Lua syntax, yes?

Yes

OK. Again, the terminology is over my head. Is the handle vs already available or does it need to be referenced/defined in some way?

The only hint I have is from the C++ headers :
//@{
	/**
	 * \brief Pass a line read from a scenario file to Orbiter for default processing.
	 * \param line line to be interpreted
	 * \param status status parameters (points to a VESSELSTATUSx variable).
	 * \note This function should be used within the body of \ref VESSEL2::clbkLoadStateEx.
	 * \note The parser clbkLoadStateEx should forward all lines not recognised
	 *   by the module to Orbiter via ParseScenarioLineEx to allow processing of
	 *   standard vessel settings.
	 * \note clbkLoadStateEx currently provides a VESSELSTATUS2 status definition.
	 *   This may change in future versions, so status should not be used within
	 *   clbkLoadStateEx other than passing it to ParseScenarioLineEx.
	 * \sa VESSEL2::clbkLoadStateEx
	 */
	void ParseScenarioLineEx (char *line, void *status) const;
	//@}

So you must take the second parameter from clbk_loadstateex, whatever the name you give it, and pass it to the parse_scenario_line_ex function, like in the example I showed.

OK. I'm unfamiliar with the inner workings of the code and I don't understand what you mean by "no references" being a problem. Is that something you can explain in plain language? Just curious.

I'm not sure but I can try ^^
CreateVariableDragElement in C++ takes a pointer to an "external drag control parameter" that is usually the animation state of a moving control surface, so that when the animation changes the drag function automatically uses the new value.
There is no valid way from the C++ side to get a pointer to a Lua variable so we must use another method.
If you want the details, the create_variabledragelement implementation allocates a memory block that will be used to store the "external drag control parameter", and pass it both to CreateVariableDragElement and as a return value to the Lua script. Then from Lua when you use the "set" command, it updates the memory block and the drag function performs its calculation with the correct value.

Are you sure your afterburner variable is a string and not a boolean? If not you must convert it with tostring(afterburner)
Oh bother, of course.

I'm actually having troubles with the oapi_init.lua file you provided - Orbiter crashes on start and I don't believe it is anything in my script. I'll try to copy and paste it again.

I placed every ScriptVessel callbacks behind lua_pcall calls to intercept errors and prevent CTD. The last error should pop at the bottom of the screen. Here are the DLLs if you want to test against OpenOrbiter : luavessel2.zip

Opps, this does not play well with CI tests...

OK, I was suffering from a self-inflicted code issue. To summarize my findings:

clbk_savestate(scn) works. Booleans must be converted to strings using the Lua tostring() function of course.
clbk_loadstateex(scn, vs2) works and there is no problem with the oapi_init.lua file you provided to implement the functions. Booleans can only be read in as strings and must  be converted back to into booleans "if boolean == "true" then boolean = true", etc.. This is known Lua behavior, but probably should be mentioned somewhere in the documentation.

So I am able to open a scenario with my Script vessel, fly it and change the animation and other states, exit Orbiter, and resume with the (Current state).scn file and pick up the simulation right where I left off. Progress! Well done!
I have seen your posts regarding LuaL_error and the annotations and it looks very promising. I will need to look into what you are doing there in more detail as I tend to make seemingly innocuous changes in my code which actually cause some conflict and the script simply starts crashing, and I have to do start commenting code until I isolate the issue. I can do it but it is aggravating. If you have figured out some way to get some more verbosity about errors to us that is great news.
I will now start looking at detecting keypress events with clbk_consumebufferedkey and see if I can get that working with my add-on.

I confess I don't quite know how to test clbk_consumebufferedkey(kstate) and I don't really know what that does relative to clbk_consumedirectkey(kstate).

clbk_consumedirectkey is called every frame with the whole keyboard state.
clbk_consumedirectkey is called only when you press/release a key. It provides the key, event and the whole keyboard state so that you can test for modifiers (ctrl, alt, shift...)

I have seen your posts regarding LuaL_error and the annotations and it looks very promising. I will need to look into what you are doing there in more detail as I tend to make seemingly innocuous changes in my code which actually cause some conflict and the script simply starts crashing, and I have to do start commenting code until I isolate the issue. I can do it but it is aggravating. If you have figured out some way to get some more verbosity about errors to us that is great news.

Did you test with the DLLs from luavessel2.zip posted above? It includes the onscreen error reporting.
Thanks for your effort :-)

clbk_consumedirectkey is called every frame with the whole keyboard state. clbk_consumedirectkey is called only when you press/release a key. It provides the key, event and the whole keyboard state so that you can test for modifiers (ctrl, alt, shift...)

You refer to consumedirectkey twice - I think one of these was meant to be consumebufferedkey? As I mentioned, clbk_consumedirectkey works fine, but I don't know what is different about clbk_consumebufferedkey or if I can test it with my current vessel script.

Did you test with the DLLs from luavessel2.zip posted above? It includes the onscreen error reporting.

I just downloaded and replaced those DLLs. I am getting a bunch of errors related to code in separate Lua modules that complain that thruster group names are undefined/nil. Indeed, they are not defined in the module, but they are defined in the main Lua script that calls those modules. I am not sure if this is actually an error or an artifact of my using Lua module files.

Thanks for your effort :-)

Thanks for all of yours! We should be able to make some real add-ons for Orbiter now with Lua script thanks to your efforts.

You refer to consumedirectkey twice - I think one of these was meant to be consumebufferedkey? As I mentioned, clbk_consumedirectkey works fine, but I don't know what is different about clbk_consumebufferedkey or if I can test it with my current vessel script.

Oops, the clbk_consumebufferedkey is call on key press. You could for example set your afterburner variable to true on a keydown event and false on keyup, no need to check everyframe.

I just downloaded and replaced those DLLs. I am getting a bunch of errors related to code in separate Lua modules that complain that thruster group names are undefined/nil. Indeed, they are not defined in the module, but they are defined in the main Lua script that calls those modules. I am not sure if this is actually an error or an artifact of my using Lua module files.

If errors are showing then your code is not executed completely. Are you defining the group names as local or global variables? I don't think a module's local variables are visible from it's "required" modules.

You could for example set your afterburner variable to true on a keydown event and false on keyup, no need to check every frame.

I might want to switch my implementation from clbk_consumedirectkey to clbk_consumebufferedkey then. I noticed that I get strange behavior if I hold the key down.

If errors are showing then your code is not executed completely. Are you defining the group names as local or global variables? I don't think a module's local variables are visible from it's "required" modules.

I have my air-breathing engine code in that module.The thruster group names are global. The code runs correctly. I wouldn't have any thrust if the jet engine module wasn't working properly. It requires a main and a dummy thruster to take the throttle input and pass it into the engine module. Not sure how to code the module to avoid this. I suppose I can just put the functions into my main code but it rather clutters things up.

I have my air-breathing engine code in that module.The thruster group names are global. The code runs correctly. I wouldn't have any thrust if the jet engine module wasn't working properly. It requires a main and a dummy thruster to take the throttle input and pass it into the engine module. Not sure how to code the module to avoid this. I suppose I can just put the functions into my main code but it rather clutters things up.

Definitely use a module. You can PM me your code on OF if you want, I can take a look at what's going on.

You could for example set your afterburner variable to true on a keydown event and false on keyup, no need to check every frame.

I suppose I need some clarification on the terminology 'keydown' and 'keyup' - I am assuming that 'keydown' is when the key is pressed, and 'keyup' is when it is released? So it basically makes the key act like a momentary switch?

I have my air-breathing engine code in that module.The thruster group names are global. The code runs correctly. I wouldn't have any thrust if the jet engine module wasn't working properly. It requires a main and a dummy thruster to take the throttle input and pass it into the engine module. Not sure how to code the module to avoid this. I suppose I can just put the functions into my main code but it rather clutters things up.

Definitely use a module. You can PM me your code on OF if you want, I can take a look at what's going on.

I may do this, but with your recent code improvements my vessel code is getting simpler, and I am starting to see some inefficiencies in my code. I think I could pass thruster names and variables without using globals, but I'd have to rewrite a lot of code. Let me clean up my code as best as I am able, and then maybe I'll take you up on your offer to look at it.

Ok, I see it now. clbk_consumebufferedkey works nicely. clbk_consumedirectkey seems to be sensitive to how long one holds the key down, which I think is expected behavior. clbk_consumebufferedkey toggles the animations more smoothly and consistently.

I have encountered a problem with the scenario_line_match functions - I thought they were reading my variables from the scenario file but apparently they were not and my initialization values were being used.
If I start a simulation and exit, the variables are written correctly to the scenario file. But if I restart with that scenario file, it won't read the values. The scenario file seems to be 100% correct, so I am rather confused as to where the problem is. I am considering trying to parse the scenario file using Lua methods.
Here is the code and the scenario file:
function clbk_loadstateex(scn, vs2)
    match = {}
    for line in scenario_lines(scn) do
        if scenario_line_match(line, "afterburner %s", match) then
            if match.res[1] == "true" then
                afterburner = true
            elseif match.res[1] == "false" then
                afterburner = false
            end
        elseif scenario_line_match(line, "gear_anim_state %d", match) then
            gear_anim_state = match.res[1]
        elseif scenario_line_match(line, "gear_anim_status %d", match) then
            gear_anim_status = match.res[1]
        elseif scenario_line_match(line, "canopy_anim_state %d", match) then
            canopy_anim_state = match.res[1]
        elseif scenario_line_match(line, "canopy_anim_status %d", match) then
            canopy_anim_status = match.res[1]
        else
            vi:parse_scenario_line_ex(line, vs2)  -- unrecognised option - pass to Orbiter's generic parser
        end
    end
end

BEGIN_URLDESC
CurrentState_img
END_URLDESC

BEGIN_ENVIRONMENT
  System Sol
  Date MJD 51983.4234207768
  Help CurrentState_img
END_ENVIRONMENT

BEGIN_FOCUS
  Ship Mirage2000Script
END_FOCUS

BEGIN_CAMERA
  TARGET Mirage2000Script
  MODE Extern
  POS 4.000000 0.000000 0.000000
  TRACKMODE TargetRelative
  FOV 40.00
END_CAMERA

BEGIN_HUD
  TYPE Surface
END_HUD

BEGIN_MFD Left
  TYPE Surface
  SPDMODE 1
END_MFD

BEGIN_SHIPS
Mirage2000Script:Mirage2000Script
  STATUS Orbiting Earth
  RPOS 4009379.065 4951024.800 218043.284
  RVEL 350.1001 -236.8856 319.5351
  AROT -65.376 -60.681 30.346
  VROT 0.1196 0.9666 0.0214
  RCSMODE 0
  AFCMODE 7
  PRPLEVEL 0:0.100000
  THLEVEL 0:0.991926 1:1.000000 2:0.000174
  NAVFREQ 94 524
  afterburner true
  gear_anim_state 1.0
  gear_anim_status 0
  canopy_anim_state 0.0
  canopy_anim_status 0
END
END_SHIPS

If I start a simulation and exit, the variables are written correctly to the scenario file. But if I restart with that scenario file, it won't read the values. The scenario file seems to be 100% correct, so I am rather confused as to where the problem is. I am considering trying to parse the scenario file using Lua methods.

Is it only the states that are not loading properly?
%d matches only integers, you need to use %f for floats :
elseif scenario_line_match(line, "gear_anim_state %f", match) then

Is it only the states that are not loading properly? %d matches only integers, you need to use %f for floats

I corrected them to %f for the floats but it doesn't change the behavior. I can initiate a simulation, fly around a bit, and then exit. The variables are written to the scenario file correctly. When I go to restart the scenario, it crashes. Any scenario that has these variables assigned to the vessel will crash on open. I can't really see anything with the ScriptMFD module that annotates the errors as it is buried in the warnings about uninitialized thruster names in my Lua modules.

I will change the %b option to accept true/false strings so you can get rid of this logic :
            if match.res[1] == "true" then
                afterburner = true
            elseif match.res[1] == "false" then
                afterburner = false
            end

Thanks!

I found my self-inflicted code issues. As far as I can see keypress detection and reading/writing vessel status to/from the scenario files all works flawlessly. I'm happy to call issues #421 and #422 resolved whenever these changes get released.
I'm flying my all-Lua vessel add-on with my all-Lua autopilot MFD on a great circle mission, and I can save it and resume it without any problems! Well done! Thanks for all of your hard efforts!
If there is anything else I can test for you I'm happy to help out.

Is this ready for Merge ?

Is this ready for Merge ?

Can we wait till sunday?
I still got some work that I'd like to push but I don't know if it'll be done by then.
If it's not ready by sunday, we can do without it for now (it's mostly panel2d/VC stuff)
I'll squash everything when it's OK to be merged, 30+ commits is a bit overkill.

Yes of course. I noticed PR is no longer 'Draft' thus the question. Take all the time you need no rush. Give it 'Ready for Merge' when ready.

Took longer than expected because I had to set up a test vessel for the virtual cockpit/2D panel work. It's a rewrite of the DeltaGlider in Lua, quite incomplete at the moment but good enough to test the new API.
There is a now also a StrictMode option available for ScriptVessel to help in debugging.

I squashed the whole 44 commits into 2 to clean up the history. One for the C++ part, and one for the Lua vessels provided as examples in the SDK.
I'm not sure how to handle the documentation with the current Latex work being done.
Apart from that, it should be good to merge.

I see there was some docking update in the main branch, let me sync this one to take it into account

I think I'm finally done with this one. You can merge it if you're OK with it.


Loading funds is super slow

wanted to buy this crazy nft :D

waited around 1 minute till it loaded

some other errors

polkadot hub and ksm chains took the most amount of time

cc @vikiival Talisman had indexer for balances, maybe we can ask them to reuse? or find otherway around

cc @vikiival Talisman had indexer for balances, maybe we can ask them to reuse? or find otherway around

Asked, in other case we can cook it with workers https://github.com/kodadot/workers/issues/254

yeah, it will also help loading the auto teleport


No documentation for resource job_template_launch

The resource job_template_launch exist in this provider but I don't see anything in the documentation about this.

should resource awx_job_template_launch be able to launch awx_workflow_job_template ?


I want to run mota.py but I don't have trackers, where did you complete the mota test?


Error with guzzle after installing the laravel-printing

Description
I've installed laravel-printing using composer. In my application i use GuzzleHttp\Client, but after the installation i get back this error when i'm using it.
Error
Interface 'Psr\Http\Client\ClientInterface' not found
If i remove laravel-printing, it works fine.
Context

Laravel Printing version: [1.3]
Laravel version: [7.29.2]

Not an issue with v2. Feel free to PR a fix to v1.x if needed.


Re-enable test_run_workflow_does_not_deserialize

Fixes https://github.com/AgnostiqHQ/covalent/issues/1127

[x] I have added the tests to cover my changes.
[x] I have updated the documentation and CHANGELOG accordingly.
[x] I have read the CONTRIBUTING document.

The test passes if covalent dispatcher server is running, but the CI didn't start it, thus the test failure.

Perhaps a few more mocks could fix this, looking...

Could appreciate a review,
CC: @kessler-frost @santoshkumarradha @AlejandroEsquivel

@kessler-frost @cjao could appreciate a re-review...


Bug - SSL Library Error - BUF_MEM_grow:malloc failure

Hi,
Our product (32-bit process) uses OpenSSL third-party libraries for EAP protocols. During the debugging of a customer issue in PEAP protocol, we got to understand that SSL_Accept has returned failure.
STATE_HANDSHAKE SSL_ERROR_SSL error retrun code [1] and peak error translate code [39] and err_get_reason [1048] // Error code, Error cases(Reason)
8900:error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca:.\ssl\s3_pkt.c:1235:SSL alert number 48 // Error Queue
SSL_ERROR_SSL Handshake failure
On further analyzing, we found that handshake failure is due to the "BUF_MEM_grow : malloc failure" in the server after the 'n' number of authentications.
The memory buffer gets completely allocated and handshake is failed when we tried to allocate the memory.
STATE_HANDSHAKE SSL_ERROR_SSL error retrun code [1] and peak error translate code [2] and err_get_reason [65]
7820:error:07064041:memory buffer routines:BUF_MEM_grow:malloc failure:.\crypto\buffer\buffer.c:122:
SSL_ERROR_SSL Handshake failure
Total memory that is installed on the server: 4GB
Process memory size when the issue occurs: 360 ~ 370 MB
Operating System Version - Windows Server 2008
OpenSSL version - 0.9.7e, 1.0.0s
Please let us know if there are known issues related to the same or let us know if you require further information for your debugging.
Regards,
Nomalatha A
Note: This issue was highlighted earlier in RT as https://rt.openssl.org/Ticket/Display.html?id=4657

https://www.openssl.org/policies/releasestrat.html

Thank you for sharing the Release Strategy. I understand that 0.9.7e and 1.0.0s is no longer supported.
The issue that I have reported above is also observed in OpenSSL version 1.0.2.h
Since our customer has reported the issue again recently, in order to raise GIT ticket, I have just copied and pasted the old RT ticket contents. Now I have updated to reflect OpenSSL 1.0.2h version as well in the original description.

How do you know the memory is allocated?  How do you know that you don't have a pointer-reuse problem? How do you know that you don't have a "we didn't free this resource" problem?
There is not sufficient information for us to help you debug this.

Thank you for your response and we would definitely provide the requested information in a few days.
Actually this particular issue cannot be simulated in our Engineering Lab and only occurs in our customer site. Therefore we need to add more instrumentation to our build and then collect the logs from the customer site and get back to you with our log analysis by answering your questions.

You should spend some time trying to see what your application is doing as well.  There is no promise that, if you send a backtrace, anyone here can or will debug the situation.   The best way forward is to make a small reproducible test case.  Yes this is very difficult.  So is long-distance debugging for free.

One thing you can do is to check to see what parameters were passed to BUF_MEM_grow when it fails to see if it gets passed an abnormal parameter or if it is sane and your system is just running out of memory.

As you have suggested, we will try to create a small reproducible test case and narrow down the problem which will help us get to the root cause. During the process of debugging, if we have any queries in OpenSSL we would like to get it clarified by this ticket. Since the issue faced by our end customer is of high priority, we are working on it actively and I would request you to keep this ticket open for a few days till we get to the root cause.

it's been three weeks with no update.  closing ticket.


⚠ Fortnite has degraded performance

In 395b350, Fortnite (https://fortnite.com) experienced degraded performance:

HTTP code: 200
Response time: 1580 ms

Resolved: Fortnite performance has improved in ae9c8a4.


PLT-7816: Create System Console Page for Data Export Config

Summary
Added System Console integration for message export. The "Run Message Export Job Now" button works, but the job that it starts currently fails, due to a bug in the PLT-7503 implementation. I'll fix that bug on the mattermost-server/PLT-7503-rewrite branch.
NOTE: Please do NOT merge this PR. It adds configuration for functionality that doesn't yet exist in master.
Ticket Link
https://mattermost.atlassian.net/browse/PLT-7816
Checklist

[x] Has enterprise changes
[x] Has server changes
[x] Has UI changes
[x] Includes text changes and localization file (.../i18n/en.json and .../webapp/i18n/en.json) updates

Setup Test Server label detected. Spinmint test server created if build succeeds (checks pass and no conflicts with base branch).

Spinmint test server created at: https://i-03027f2b5df6ebbbb.spinmint.com
Test Account 1: Email<EMAIL_ADDRESS>| Password: passwd
Test Account 2: Email<EMAIL_ADDRESS>| Password: passwd
Instance ID: i-03027f2b5df6ebbbb

Spinmint test server destroyed

Have started on text reviews, but waiting for the server PRs to be merged before I can try this.

@grundleborg @hmhealey please take another pass at this when you get a moment


run unet.py failed

I downloaded the repository and tried running unet.py, then I met the err
IOError: [Errno 2] No such file or directory: '../npydata/imgs_train.npy'

you can download the data from http://pan.baidu.com/s/1qY3ARA8 ,password:2sy3


Add more metadata to be returned by class UnstructuredFileLoader()

Add more metadata to be returned by class UnstructuredFileLoader().
This PR adds more metadata that could be useful when dealing with typical document file types like docx, pdf, odt, etc. The additional metadata, on top of the existing 'source' info, are:

file type,
file modified date and
file creation date.

As such, the metadata dict looks like this:
metadata = {
            "source": self.file_path,
            "filetype": detect_filetype(filename=self.file_path).name,
            "file_mod_date": self._get_file_mod_date(),
            "file_create_date": self._get_file_create_date(),
}

The file type info is extracted using the native unstructured method from unstructured.file_utils.filetype import detect_filetype. The file modified date and creation date are extracted using normal os methods, and its process is abstracted to internal methods  _get_file_mod_date() and _get_file_create_date().
Let me know if you guys find these new metadata useful. I personally had a need for it in my use case. As such, just contributing here in case other people might need it too :)
@hwchase17
@eyurtsev

Hello @marcusyatim contribution much appreciated! I added some high level comments. I would love to standardize metadata handling across the various loaders! I added some comments, I think there are some really tricky design questions around metadata handling.

@eyurtsev Let me know what you think of the new changes :) happy to receive any feedback and/or edits! Thanks again!

@MthwRobinson thoughts on this?

Looks good to me! We do have plans to add some of this info to the metadata on the unstructured side as well. We can loop back here to make updates as we add new metadata fields.

@marcusyatim Hi , could you, please, resolve the merging issues and address the last comments (if needed)? After that, ping me and I push this PR for the review. Thanks!

Hi @leo-gan, sure thing, will do so asap. Thanks!

Hey @marcusyatim ! Closing because the PR wouldn't line up with the current directory structure of the library (would need to be in /libs/langchain/langchain instead of /langchain). Feel free to reopen against the current head if it's still relevant!


Ignore Early Hints (5.x)

Ignore for now to solve https://github.com/square/okhttp/issues/7430
Initial PR from https://github.com/square/okhttp/pull/7196

Yes, love this. PROOOOBABLY want this on 4.x also.

Yep, will do.


Replace api/dataserver with api/

Not yet ready for merging, comments welcome
Update AQ, Traffic and Car Parking to access historic data using the new Django API rather than  api/dataserver.
Closes #173, #178

Ability to retire /api/dataserver depends on #180

Closes #173


Add Hungarian translation

Hungarian translation (1st version)

@mandris36 Thank you so much for this translation, and I really apologise for having taken so long to review this. Everything looks great, very happy to merge this is. Once it's merged I'll update the related configurations and the Hungarian translation should be live very shortly. I will comment one more time here once it is live 😊

@mandris36 the Hungarian translation is now live on https://blackholehunter.org/, that so much once again for creating the translation, it's a fantastic addition to the site 😊


Fix several validation and presentation issues in function commands

Follow-up to #5882
There’s more checks and validations. Also better printing with progrock, making the loading vertex unfocused and pushing -help and dagger functions to the execution vertex (focused).

[...] do you think now's a good time to add a few basic integ tests for dagger call and dagger export (dagger shell is quite difficult to integ test, probably a whole separate effort)?
Can be follow-up too of course, but we'll want them soon I think.

Yes, I mean to add tests asap, but I need to prioritize Python modules rn.


⚠ Bofrost.lu has degraded performance

In 2341f9a, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 2531 ms

Resolved: Bofrost.lu performance has improved in 10f279f after 9 minutes.


Setup monorepo structure, tools, etc

[x] evaluate what solution to opt for based on our technical watch
[x] set-up the new dual build system (for both npm and CDN)
[x] set-up packages publication on npm
[x] set-up bundles publication on a CDN
[x] set-up demos using the CDN build
[ ] add all associated CI/CD configurations
[x] code formatting and linting
[x] test runner

For the first "technical watch" step, our goal with @jboileau99 was to find the best solution to provide this in a minimum amount of time.
As expected, we didn't find any ready-to-use tools or seed project matching our needs.
At the moment, we identified several inspiring projects. Most of them being "Design Systems" / UI Frameworks. For example:

Elix

very interesting but way too far from what we need

Microsoft Fast

use TypeScript, Beachball and Jest
only provide a single bundle

:x: NOK because far from our context (design system + use lots of new tools) which would require too many changes and specific development
Few projects were closer to our context and needs. The best (and well, only) one we identified at the moment was:

CleverCloud clever-components

use JS, Rollup, and Web-Test-Runner
provide CDN builds for their "Smart CDN"
not technically/only a design system per se

:heavy_check_mark: good start/inspiration, as it is close to our context and needs and use pretty much the same tools and approach
After that, I also identified other projects that could be inspiring:

patternfly-elements (see get started with unpkg)
shoelace (only provide a single bundle via jsdelivr, but has an interesting minimalistic build script)

I'll continue to investigate this on my own as I implement the following steps.

I finally ended up custom building it from scratch as the above examples didn't fit our needs.

Here is a great introduction to what I intend to do for the last "CI/CD" step: Automatic versioning in a Lerna monorepo using Github actions

Actually, I addressed all parts of this issue in #39, so I should have replaced "partially address" with "close" before merging it :smile:


TRL SFTTrainer Examples

Simplified the SparseML Trainer to be a barebones class definition, everything is handled by SessionManagerMixIn. Removed a bunch of old code from loading recipes, as this is handled by SparseAutoModelForCausalLM
Added new SFTTrainer class which adds our mix-in to trl's SFTTrainer. The only added code here is to add support for passing in a tokenized dataset to SFTTrainer
Added examples of using SFTTrainer for sparse finetuning, both with out dataset preprocessing and TRL's dataset preprocessing

Asana ticket: https://app.asana.com/0/1201735099598270/1206486351032763/f
Testing
See examples in src/sparseml/transformers/finetune/examples

Thanks Sara - this looks really nice
Are there any other features we should flex? I am thinking we might want to look at:

FSDP
Distillation


[#2PlaysAMonth]:  Code Snippets

What's the Play about?
Code Snippets
Description: A collection of code snippets which allows you to copy and paste your code snippet and lets you download it as an image to share!
What ReactJS concept will be used to create this play?
Context API, Hooks, React Router.
Code of Conduct

[X] I agree to follow this project's Code of Conduct

Very cool idea.


Implemented Certificate Generator

🛠 Fixes Issue (Number)
Fixes #831
👨💻 Changes proposed and Brief Description

Implemented Certificate Generator using Javascript

✅ Check List (Check all the applicable boxes)

[x] My code doesn't break any part of the project
[x] This PR does not contain plagiarized content.
[x] My Addition/Changes works properly and matches the overall repo pattern.
[x] The title of my pull request is a short description of the requested changes.

Screenshots

Review Requested by @swapnilsparsh
Kindly add labels: GSSoC and Level 3 . Thank you!


Add build status badge to README

Since you've added Travis support to your repository, I suggest you also add a "build status" badge to your README

Actually it is in over there.


Default subtitle flavor configuration

This patch provides a default subtitle flavor configuration which should work with many installations out of the box.
More than that, it should never cause any trouble because the subtitle editor by itself is still hidden by default.

The captions/source+<lang> flavor idea comes from us. As these subtitles are still source files in the sense that they are tight to the source video and need to be trimmed together with the video, we applied the same idea of having a captions/delivery+<lang> flavor. What do you think of using that as the new default for Opencast?

I think that's a great idea for the future.
But this isn't changing the default, and the captions/source+<lang> that are removed are just comments.
I could already set that now, but at the moment, adopters are more likely to use captions/vtt.
If we want captions/source+<lang> to actually work (+1 for that), we also need to change more things in Opencast. In particular, the default workflow would actually need to cut subtitle files and we would need to publish only the ones we cut.

My original intention for this was mostly to provide a default configuration which works against the data on develop.opencast.org so that users can actually test the subtitle editor easily. But #828 provides a separate configuration for that anyway.
I see that just having a default set in here seems surprisingly controversial. That's why I'm declining this for now. But I'm going to start a broader conversation of how subtitles should work in Opencast and how we should apply flavors and tags. Once that discussion comes to a conclusion, we can revisit this.


Container black-list PUIDs and Extensions

Black-list of container file types that we can identify DROID won't analyse. There will need to be further analysis done on these files in any collection. E.g. RAR.

Taken care of if we specify in Blacklist.cfg.


Cache middleware broke in 2.2.*

Honojs cache middleware broke since 2.2.*
Using the middleware and publishing or running dev in wrangler does not give any errors. Once the route is called it will return a 500 with Trace: This context has no ExecutionContext in the worker log.
Running a very simple honojs Cloudflare Workers will reproduce this error:
import { Context, Hono, Next } from 'hono';
import { cache } from 'hono/cache'

const app = new Hono();

app.get('*', cache({ cacheName: 'my-app', cacheControl: 'max-age=3600' }))
app.get('/', (c) => c.text('Pretty Blog API'));

app.fire();

Hi @alinoil !
We've fixed this issue and released a new version "v2.2.3". Try it and let us know if you have any more problems.


Anchors

The ability to add links in the drawing which send you to other parts of the drawing, much like anchors in HTML. Rather than a series of lines, the user (especially when presenting) could click a link that moves the view to the resulting location.

It would be great if (much like html) these anchors could be part of the URL. This way, you could place a link on any external document/website that would take you to a specific point in the referenced draw.io diagram.

I am using google bookmarks and i have huge maps it would be great to have a changing url with the location so i can come back to the exact position in the map

for the 1st request above custom links might help:
https://desk.draw.io/support/solutions/articles/16000080137-work-with-custom-links


De-noising Autoencoder module

Hello @swapsha96! Thanks for opening this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file mirapy/__init__.py:

Line 26:1: W391 blank line at end of file

In the file mirapy/autoencoder/models.py:

Line 17:1: W391 blank line at end of file

I have added a Jupyter notebook as tutorial here.


Upgrade to rasa 2.4

Issue https://github.com/RasaHQ/financial-demo/issues/105

[x] Upgrade to rasa 2.4, with  PR https://github.com/RasaHQ/rasa/pull/7949
[x] Deploy to VM and do an integration test

@ArjaanBuijk looks good to me.


fix broken build cargo test

It looks likes the test was not updated to match an API change.

Ah yes, I missed that, thank you!


selected chapter should be highlighted

Setting Cookie and fetching the cookie to highlight the selected book.

As i have checked on the branch react_navbar and still  am not able to see the functionality working. I found another issue here as well. When I start app first time and click on the chapter tab and select chapter then book name doesn't appear in the navbar button. Please have a look at this issue.

implemented with mobx


Table naming pattern

@constantinpape
Sorry, small thing, but for next run, could you maybe change this:
20200417_132123_311_cell_table_serum_IgG.xlsx*
20200417_132123_311_image_table.xlsx*
20200417_132123_311_well_table.xlsx*

to
20200417_132123_311_cell_serum_IgG_table.xlsx

would be more consistent naming scheme and a bit easier to parse.
that is:
PLATENAME_CONTENT_table.xlsx

Yes, makes a lot of sense.

Implemented in #130. Gonna close once it's merged.


Test positioning named widgets (via widget_player) on DMD

All 4 differently positioned widgets end up centered on the virtual dmd widget.

I've discovered at least part of the problem. The method set_position() called from here assumes widgets have a parent, but this may not be true for widgets added via a widget_player config. We're also currently not handling the AttributeError in the try block, so the 'NoneType' object has no attribute 'width' and 'NoneType' object has no attribute 'parent' errors are not being caught. Python was also complaining about accessing the dict with self.config['x'] vs self.config.get('x'). Beyond widget.parent`, is there another way to find the parent DMD?


Enable compilation and runtime roundtrip tests for microbenchmarks

@pzread you can consider to add microbenchmarks to this folder. For convolution cases, we don't have to start with mhlo level. We can add linalg_conv_xxx.mlir files instead.

Wait. I think this will by default put all microbencmarks to tests and gotten invoked when we ctest? Can we make these opt-in and not run by default when running tests?

Yes, you're right. It's sounds good to me. How do I make them opt-in and not run by default?

Not sure ctest supports optionality.. Could use a CMake option like IREE_BUILD_MICROBENCHMARKS to guard against this I think.

Wait. I think this will by default put all microbencmarks to tests and gotten invoked when we ctest? Can we make these opt-in and not run by default when running tests?

Just curious why not run these tests? As far as I understand, they are just making sure these compile and run, and it should be a small fraction of the  time taken by other tests (like the model tests). So I think it is OK to be added by default?

After thinking a while, maybe it's fine to put them to default ctest? The purpose is to test compilation and runtime for the ops. We're not running microbenchmarks in ctest.

Time to run these microbenchmarks is indeed one concern. For stable benchmarks we tend to include large-shaped ops. Also we want to consider many variants. It can adds up quickly. Looks right now the existing ones is taking ~100s, while model tests taking ~150s. So they are at similar levels of latency already. I think we are looking to include further stuff here.
More importantly (micro)benchmarks are not tests. Tests are meant for correctness (including those model tests). Each checks a different feature or from a different angle. Typically we don't check in multiple tests covering the same thing. Microbenchmarks actually does that by just focusing one single op and varying its shape. There is little additional coverage here to warrant always running them in each ctest invocation for everybody during daily development.
I do see the needs of keeping these IRs up to date and able to compile though. (Given that there is no verifying the output at all right now, I'm not sure we need to actually run them.) It happens at a quite low frequency and only after some integration. If we want to guard against such stale IR cases, using CI is enough for it IMO. So making it optional and default to off and enable for CIs would cover that, and we don't need to pay the cost during daily development after each ctest invocation. :)

Time to run these microbenchmarks is indeed one concern. For stable benchmarks we tend to include large-shaped ops. Also we want to consider many variants. It can adds up quickly. Looks right now the existing ones is taking ~100s, while model tests taking ~150s. So they are at similar levels of latency already. I think we are looking to include further stuff here.
More importantly (micro)benchmarks are not tests. Tests are meant for correctness (including those model tests). Each checks a different feature or from a different angle. Typically we don't check in multiple tests covering the same thing. Microbenchmarks actually does that by just focusing one single op and varying its shape. There is little additional coverage here to warrant always running them in each ctest invocation for everybody during daily development.
I do see the needs of keeping these IRs up to date and able to compile though. (Given that there is no verifying the output at all right now, I'm not sure we need to actually run them.) It happens at a quite low frequency and only after some integration. If we want to guard against such stale IR cases, using CI is enough for it IMO. So making it optional and default to off and enable for CIs would cover that, and we don't need to pay the cost during daily development after each ctest invocation. :)

These are indeed meant for correctness (and by the above argument should run CIs). The point is these micro benchmarks should always compile + run. Right now these are not "testing the result" cause they are bigger size inputs and outputs and testing the result for these sizes is hard. I think it is useful to have the test compile + run (run is just to make sure it doesnt say segfault at runtime or something like that) even if these tests dont have correctness included.
100s is a lot actually, but it is not that much. I think its worth having these.

Interestingly @MaheshRavishankar, @hanhanW and I had exactly the same discussion a few days ago :).  I think I align with what @antiagainst said: testing and benchmarking should be "independent". Otherwise, testing will impose unnecessary runtime constraints on what we can benchmark. Running the benchmarks won't provide too much value from a testing perspective since we are not validating the output. Yes, we would know if the benchmark is crashing or not but that's only marginal information. We can mess up the IR in many ways that won't lead to a runtime crash. What I've seen in other projects is that benchmarks are accompanied by representative e2e tests where the output is checked and that run within a more reasonable time budget. Even e2e tests may become problematic: https://discourse.llvm.org/t/check-mlir-times-examples-standalone-testing-time/6073 :)
I think enabling benchmarks with a cmake flag could be a good first step, as suggested above.
Hopefully that helps!

Interestingly @MaheshRavishankar, @hanhanW and I had exactly the same discussion a few days ago :). I think I align with what @antiagainst said: testing and benchmarking should be "independent". Otherwise, testing will impose unnecessary runtime constraints on what we can benchmark. Running the benchmarks won't provide too much value from a testing perspective since we are not validating the output. Yes, we would know if the benchmark is crashing or not but that's only marginal information. We can mess up the IR in many ways that won't lead to a runtime crash. What I've seen in other projects is that benchmarks are accompanied by representative e2e tests where the output is checked and that run within a more reasonable time budget. Even e2e tests may become problematic: discourse.llvm.org/t/check-mlir-times-examples-standalone-testing-time/6073 :)
I think enabling benchmarks with a cmake flag could be a good first step, as suggested above.
Hopefully that helps!

Well there is performance tracking and there is testing. They are indeed independent, but making sure things that are needed for performance tracking are "not broken" should very much be part of testing. We could maybe add this as part of post-submit testing if 100s of testing is too onerous (no one needs/should run the test all the time. You probably need to run it once locally to verify, or just let CI run it which is what I do, so 100s is very doable). Right now we are relying on post-submit to catch places where things are broken and either revert or fix forward on those errors. We could do that for these tests too if there is a place in the post-submit CI these could slot in.
Re: full correctness testing, of course that would be better, but is hard to do for large tensors. At least a runtime crash tells you something is broken (like maybe too much stack allocation, etc.). Hopefully other smaller unit tests are enough to catch correctness issues.
I still dont see a reason to not have these tests checked in. Maybe post-submit is an option.

I understand that putting this to default adds overheads for everybody during daily development. It makes sense to have an option to control it. People who care it can turn on the option themselves. We can ask CI to test it.
The current revision only enables compilation, but not runtime. It's okay to me because I believe they are all covered in our unit tests and integration tests. We can add runtime tests to CI if we run into issues.
Do we have preference to run it on which CI bot?
I could not find how to put it to other presubmit bots, so I temporary add it to linux/x86-swiftshader-asan.

I still dont see a reason to not have these tests checked in. Maybe post-submit is an option.

Having them checked by CI is fine to me; either presubmit or postsubmit works. I was saying it's better to allow the optionality so that folks can choose to avoid running these in the daily development workflow, as they are not the same as the other tests and we have correctness coverage for these ops in e2e/ directory. :)

I still dont see a reason to not have these tests checked in. Maybe post-submit is an option.

Having them checked by CI is fine to me; either presubmit or postsubmit works. I was saying it's better to allow the optionality so that folks can choose to avoid running these in the daily development workflow, as they are not the same as the other tests and we have correctness coverage for these ops in e2e/ directory. :)

Understood. I guess personally I dont run into the issue since I have vscode setup so I can run different test suites. So I rarely run all tests (mostly cause we have a resnet test in the e2e/models that takes as much time as all other tests combined in debug builds)


Attention maps issue

Hello. I am trying to implement your code and faced weird behavior of Attention Maps (predMaskC and predMaskN). I am using your pretrained model. I  also tried torchvision 0.2.0 and torch 0.4.0, but it didn't help. Do you have any suggestions?

Do your python version is 2.7 ?

Is your python version 2.7 ?

I have tried both python versions (2.7 and 3.6), same result.
My code for your model inference:
`
img_orig = skimage.io.imread('rgb_kitti.png')
lidar = skimage.io.imread('lidar_kitti.png').astype(np.float32)
sparse = lidar * 1.0 / 256.0
binary_mask = np.where(sparse > 0.0, 1.0, 0.0)
img = img_orig.transpose(2, 0, 1)
left = torch.FloatTensor(img.reshape(1, img.shape[0], img.shape[1], img.shape[2])).cuda()
sparse = torch.FloatTensor(sparse.reshape(1, 1, sparse.shape[0], sparse.shape[1])).cuda()
mask = torch.FloatTensor(binary_mask.reshape(1, 1, binary_mask.shape[0], binary_mask.shape[1])).cuda()
model = s2dN(1)
model = nn.DataParallel(model, device_ids=[0])
model.cuda()
state_dict = torch.load('depth_completion_KITTI.tar')['state_dict']
model.load_state_dict(state_dict)
model.eval()
with torch.no_grad():
outC, outN, maskC, maskN = model(left, sparse, mask)
tempMask = torch.zeros_like(outC)
predC = outC[:,0,:,:]
predN = outN[:,0,:,:]
tempMask[:, 0, :, :] = maskC
tempMask[:, 1, :, :] = maskN
predMask = F.softmax(tempMask)
predMaskC = predMask[:,0,:,:]
predMaskN = predMask[:,1,:,:]
pred1 = predC * predMaskC + predN * predMaskN
pred = torch.squeeze(pred1)
pred = pred.data.cpu().numpy()
pred = np.where(pred <= 0.0, 0.9, pred)
pred_show = pred * 256.0
pred_show = pred_show.astype('uint16')
res_buffer = pred_show.tobytes()
img_pred = Image.new("I",pred_show.T.shape)
img_pred.frombytes(res_buffer,'raw',"I;16")
img_pred.save('img_pred.png')
`

I also have tried different images from Kitti dataset, and predMaskC is always practically empty. As a result there is no impact from Color Pathway Dense Depth in a final one.

The value of pixel in the mask is from 0 to 1, you should multiply 255 for visualization.

The value of pixel in the mask is from 0 to 1, you should multiply 255 for visualization.

The problem is not in visualization, matplotlib allows showing images both in the range of 0 to 255 and 0 to 1. The main issue is that the Mask of Color Pathway is almost all consists of zeros, while the Mask of Normal Pathway consists of ones.

Did anyone face such issue?

Sorry, I don't know. The code I test in my machine is actually fine.
On Mon, Feb 10, 2020 at 10:45 PM 421psh<EMAIL_ADDRESS>wrote:

Did anyone face such issue?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/JiaxiongQ/DeepLiDAR/issues/25?email_source=notifications&email_token=AJANJRGL6BV334XK6ECFBG3RCFSAVA5CNFSM4KQ3R3S2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELIYLIQ#issuecomment-584156578,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJANJRBVBTXHD5M6KGCKMD3RCFSAVANCNFSM4KQ3R3SQ
.

@JiaxiongQ Thank you for your responsiveness. Finally, after neat preprocessing on the input data and following all requirements, I actually managed to get right Depth Map. Result is great, but I am wondering are you  going to adapt the code to python 3 and up to date versions of torch and torchvision?

Great! This is a good suggestion, but we don't have such plan in the near
future.
On Tue, Feb 11, 2020 at 6:47 PM 421psh<EMAIL_ADDRESS>wrote:

@JiaxiongQ https://github.com/JiaxiongQ Thank you for your
responsiveness. Finally, after neat preprocessing on the input data and
following all requirements, I actually managed to get right Depth Map.
Result is great, but I am wondering are you going to adapt the code to
python 3 and up to date versions of torch and torchvision?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/JiaxiongQ/DeepLiDAR/issues/25?email_source=notifications&email_token=AJANJRGIRYIODCPDRQNFJBTRCJ64JA5CNFSM4KQ3R3S2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELL6X2Q#issuecomment-584575978,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJANJRCLUAS5TVRVTBJDO6LRCJ64JANCNFSM4KQ3R3SQ
.

Hello. I am trying to implement your code and faced weird behavior of Attention Maps (predMaskC and predMaskN). I am using your pretrained model. I  also tried torchvision 0.2.0 and torch 0.4.0, but it didn't help. Do you have any suggestions?

why my result is like this

The version of python should be 2.7
On Thu, May 14, 2020 at 8:06 PM Wen Y.K<EMAIL_ADDRESS>wrote:

Hello. I am trying to implement your code and faced weird behavior of
Attention Maps (predMaskC and predMaskN). I am using your pretrained model.
I also tried torchvision 0.2.0 and torch 0.4.0, but it didn't help. Do you
have any suggestions?
why my result is like this
[image: image]
https://user-images.githubusercontent.com/36818370/81932377-4fc95000-961e-11ea-8d4b-9f436768cf12.png
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/JiaxiongQ/DeepLiDAR/issues/25#issuecomment-628589785,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJANJRFJFR2UWGVT3ECYGJLRRPNDHANCNFSM4KQ3R3SQ
.

The version of python should be 2.7
…

I have used python 2.7 （a long time for config env） but it still like this
oh on


Elois cache eth sig verif

Cache ethereum signature verification in wasm instance. And prune the cache at block execution time.
Closes #448

Can you use log and run an instance of node to see if it works?
If this is ran in wasm it won't work, as each time it creates a new wasm instance. It does work to some extend if it uses native-only, but that's really limited and shouldn't be relied on.

I haven't looked at the PR, but @sorpaas the idea for caching is mostly for the validate, outside of the block production, which is happening at regular interval. When the txpool is full, it adds some load to revalidate.

I'm going to close this since it looks invalid after #482. Substrate should now handle the sig cache.


bug: DDL grammar

when I execute 'insert  into table xxxxx', display error
"Code: 27, e.displayText() = DB::Exception: Cannot parse input: expected '(' before: 'FORMAT JSON': at row 1 (version <IP_ADDRESS>)",  need to add a semicolon ';' to execute.

This is because ClickHouse does not support INSERT formatting using the Http interface.


OAuth20CallbackAuthorizeController can NOT get the any URL from CallbackController(spring-webmvc-pac4j 1.1.1)

CAS Server: v5.0.0.M3
Container: Tomcat 8.0, Tomcat 8.5
OAuth module can't finish the Authorize Code Flow.
OAuth20CallbackAuthorizeController got a null, when getting callback url with pac4j's CallbackController.
package org.apereo.cas.support.oauth.web;

......

/**
 * OAuth callback authorize controller based on the pac4j callback controller.
 *
 * @author Jerome Leleu
 * @since 3.5.0
 */
@Controller("callbackAuthorizeController")
public class OAuth20CallbackAuthorizeController extends AbstractController {

   ......

    @RequestMapping(path = OAuthConstants.BASE_OAUTH20_URL + '/' + OAuthConstants.CALLBACK_AUTHORIZE_URL)
    @Override
    protected ModelAndView handleRequestInternal(final HttpServletRequest request, final HttpServletResponse response) throws Exception {
        final String url = StringUtils.remove(this.callbackController.callback(request, response), "redirect:");
        // url is null
        final J2EContext ctx = new J2EContext(request, response);
        final ProfileManager manager = new ProfileManager(ctx);
        return oAuth20CallbackAuthorizeViewResolver.resolve(ctx, manager, url);
    }

   ......

}

@mmoayyed @leleuj Should this be resolved in CallbackController or OAuth20CallbackAuthorizeViewResolver?

Why is it null? Can you step through?

I just traced into CallbackController(spring-webmvc-pac4j 1.1.1). And it's callback method will always return null.
I have a little confused.
package org.pac4j.springframework.web;

......

/**
 *
This filter finishes the login process for an indirect client, based on the {@link #callbackLogic}.

 *
 *
The configuration can be defined via property keys: pac4j.callback.defaultUrl (default url after login if none was requested),
 * pac4j.callback.multiProfile (whether multiple profiles should be kept) and
 * pac4j.callback.renewSession (whether the session must be renewed after login).

 *
Or it can be defined via setter methods: {@link #setDefaultUrl(String)}, {@link #setMultiProfile(Boolean)} and ({@link #setRenewSession(Boolean)}.

 *
 * @author Jerome Leleu
 * @since 1.0.0
 */
@Controller
public class CallbackController {

    ......

    @RequestMapping("/callback")
    public String callback(final HttpServletRequest request, final HttpServletResponse response) {

        assertNotNull("callbackLogic", callbackLogic);
        assertNotNull("config", config);
        final J2EContext context = new J2EContext(request, response, config.getSessionStore());

        callbackLogic.perform(context, config, J2ENopHttpActionAdapter.INSTANCE, this.defaultUrl, this.multiProfile, this.renewSession);

        return null;
    }

   ......

}

Looks like a bug in pac4j? I am not sure why that method is designed to have a return value, and yet always returns null.

The return null is normal. Redirections or any other HTTP response is applied directly to the response by the perform method (remember pac4j fits in for many frameworks with various schemes).
And indeed, the final String url = StringUtils.remove(this.callbackController.callback(request, response), "redirect:"); does not make sense here.
If you want to get the redirection, you certainly need to get the "Location" header from the response.

Shouldnt the return type of that method be changed to be void instead? If you're always returning null, why indicate that you're trying anything? Are there use cases where null is not returned?

And, since the old behavior was to return the url: why not preserve compatibility and return that instead? rather than requiring the user to peek into "location"?

You're right, we should certainly return void, that should make more sense. Tracked here: https://github.com/pac4j/spring-webmvc-pac4j/issues/8
For Spring Web MVC, null is always returned. I admit it's not backward compatible, but it should not be a big deal as you should not need to get this url, the redirection is performed automatically.
Why do you need to get the redirection url?

I just found this in CallbackController(spring-webmvc-pac4j 1.1.1):
    @RequestMapping("/callback")
    public String callback(final HttpServletRequest request, final HttpServletResponse response) {

        assertNotNull("callbackLogic", callbackLogic);
        assertNotNull("config", config);
        final J2EContext context = new J2EContext(request, response, config.getSessionStore());

        callbackLogic.perform(context, config, J2ENopHttpActionAdapter.INSTANCE, this.defaultUrl, this.multiProfile, this.renewSession);

        return null;
    }

And
public class J2ENopHttpActionAdapter implements HttpActionAdapter<Object, J2EContext> {

    public static final J2ENopHttpActionAdapter INSTANCE = new J2ENopHttpActionAdapter();

    @Override
    public Object adapt(int code, J2EContext context) {
        return null;
    }
}

Seems the redirection doesn't been performed yet.

The redirection is performed here: https://github.com/pac4j/pac4j/blob/master/pac4j-core/src/main/java/org/pac4j/core/engine/DefaultCallbackLogic.java#L110

I'll run some tests later this evening and see if I can reorganize this change. Thanks for reviewing.

See if RC1-SNAPSHOT solves this. I tested it somewhat thoroughly.

Committed: https://github.com/pac4j/spring-webmvc-pac4j/commit/b9c283cacd5a5dd553c94c37b923e0defa4f1520


[DO NOT MERGE] Validate Bug 710885

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/welcome.md
:white_check_mark:Succeeded
View
Details

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Can I close this @superyyrrzz?

Yes, it's no use now.


Supporting Promise based interface in library

Any chances?
PS :- You can assign it to me, if any plans.

Libraries like bluebird offer promisify which should resolve this issue.
http://bluebirdjs.com/docs/api/promise.promisify.html
var imageDiff = Promise.promisify(require('image-diff'));

@mlmorg Can we please close this issue?


Unable to build documentation - [zh-tw] REF_NOT_FOUND

Description
I am trying to build the documentation for gitea from source. It is not working in past few releases.
gitea/docs on  release/v1.19 took 2s
❯ make build-offline
go run<EMAIL_ADDRESS>--baseURL="/" --cleanDestinationDir
Start building sites …
hugo v0.111.3 linux/amd64 BuildDate=unknown
ERROR 2023/05/03 23:15:10 [zh-tw] REF_NOT_FOUND: Ref "doc/administration/config-cheat-sheet.zh-tw.md": "/run/host/home/balki/projects/notmine/gitea/docs/content/doc/usage/push.zh-tw.md:55:358": page not found
Error: Error building site: logged 1 error(s)
Total in 267 ms
exit status 255
make: *** [Makefile:28: build-offline] Error 1

gitea/docs on  release/v1.19 took 2s
❯ git status
On branch release/v1.19
Your branch is up to date with 'origin/release/v1.19'.

nothing to commit, working tree clean

gitea/docs on  release/v1.19
❯ git log -1 --oneline
8ceb78caa (HEAD -> release/v1.19, tag: v1.19.3, origin/release/v1.19) Changelog for 1.19.3 (#24495)

Gitea Version
1.19.3
Can you reproduce the bug on the Gitea demo site?
No
Log Gist
No response
Screenshots
No response
Git Version
No response
Operating System
No response
How are you running Gitea?
N/A
Database
None

I meet this too: https://github.com/go-gitea/gitea/pull/24405/files#diff-017a68c3258bd68503600a280f83d0f0a7deb4ad3da1619343544c9497f11c84
I have no idea why it works after:
-({{< relref "doc/administration/config-cheat-sheet.zh-tw.md#repository-repository" >}})
+({{ < relref "doc/administration/config-cheat-sheet.zh-tw.md#repository-repository" > }})

Can't reproduce with:
make docs
cd docs
make server

Did you run make docs before cd docs?

Did you run make docs before cd docs?

No. Do I need to? I mean, I don't docs with make help. And what I want is "previewing" docs when editing them, not "building".

I thought it's required

I think it is because the file is missing in one language. Creating  the file fixes the problem. Not sure how make docs in src fixes the problem.
gitea/docs on  release/v1.19
❯ cp content/doc/administration/config-cheat-sheet.zh-cn.md content/doc/administration/config-cheat-sheet.zh-tw.md

gitea/docs on  release/v1.19 [?]
❯ make build-offline
go run<EMAIL_ADDRESS>--baseURL="/" --cleanDestinationDir
Start building sites …
hugo v0.111.3 linux/amd64 BuildDate=unknown

                   | ZH-CN | EN-US | ZH-TW | PT-BR | NL-NL | FR-FR | DE-DE
-------------------+-------+-------+-------+-------+-------+-------+--------
  Pages            |    35 |    90 |    33 |     6 |     6 |    17 |     8
  Paginator pages  |     0 |     0 |     0 |     0 |     0 |     0 |     0
  Non-page files   |     0 |     0 |     0 |     0 |     0 |     0 |     0
  Static files     |    27 |    27 |    27 |    27 |    27 |    27 |    27
  Processed images |     0 |     0 |     0 |     0 |     0 |     0 |     0
  Aliases          |     1 |     0 |     0 |     0 |     0 |     0 |     0
  Sitemaps         |     2 |     1 |     1 |     1 |     1 |     1 |     1
  Cleaned          |     0 |     0 |     0 |     0 |     0 |     0 |     0

Total in 274 ms

Yes. looks like missing file is the problem. When running make docs, it calls trans-copy script, which copies all missing files from English.
# This script is used to copy the en-US content to our available locales as a
# fallback to always show all pages when displaying a specific locale that is
# missing some documents to be translated.
#

Please close the issue if that is how docs need to be built. Thanks!


chore: bdd-sbhg-1585039502 to 0.0.1

chore: Promote bdd-sbhg-1585039502 to version 0.0.1

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approvers:
If they are not already assigned, you can assign the PR to them by writing /assign  in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


adding glossary

Glossary defines key terms used in the OLM book and project. This is an initial commit. The glossary should help formalize OLM terms.

/cc @kevinrizza

/lgtm


Update active transferwise variants to point to the corresponding subdirectories in transferwise/pipelinewise

To better serve Wise business and customer needs, the PipelineWise codebase needs to shrink. We have made the difficult decision that, going forward many components of PipelineWise will be removed or incorporated in the main repo.

On Slack:

https://meltano.slack.com/archives/C069A0GE129/p1730907152827889

Example from Slack:
  name: target-postgres
- pip_url: pipelinewise-target-postgres
+ pip_url: git+https://github.com/transferwise/pipelinewise.git#subdirectory=singer-connectors/target-postgres
  variant: transferwise


Can't compile for x64

Trying to compile a x64 app for Windows.
Tried passing "x64" in nodeVCBuildArgs, but is ignored and compiled as x86 anyway

Thanks for the report. Will look into it. On a sidenote, I'm 100% sure I've seen you somewhere...

I got the same issue just now as well:
D:\git\deepstream.io\nexe\src\node\4.0.0\node-v4.0.0\node.sln.metaproj : error MSB4126: The specified solution configuration "Release|X86" is invalid. Please specify a valid solution configuration using the Configuration and Platform properties (e.g. MSBuild.exe Solution.sln /p:Configuration=Debug /p:Platform="Any CPU") or leave those properties blank to use the default solution configuration. [D:\git\deepstream.io\nexe\src\node\4.0.0\node-v4.0.0\node.sln]
....> ERROR: The release executable has not been generated. This indicates a failure in the build process. There is likely additional information above.

However the configuration seems to contain the correct variable:
{ 'target_defaults': { 'cflags': [],
                       'default_configuration': 'Release',
                       'defines': [],
                       'include_dirs': [],
                       'libraries': []},
  'variables': { 'asan': 0,
                 'host_arch': 'x64',
                 'icu_small': 'false',
                 'node_byteorder': 'little',
                 'node_install_npm': 'true',
                 'node_prefix': '/usr/local',
                 'node_release_urlbase': '',
                 'node_shared_http_parser': 'false',
                 'node_shared_libuv': 'false',
                 'node_shared_openssl': 'false',
                 'node_shared_zlib': 'false',
                 'node_tag': '',
                 'node_use_dtrace': 'false',
                 'node_use_etw': 'true',
                 'node_use_lttng': 'false',
                 'node_use_openssl': 'true',
                 'node_use_perfctr': 'true',
                 'openssl_fips': '',
                 'openssl_no_asm': 0,
                 'python': 'C:\\Python27\\python.exe',
                 'target_arch': 'ia32',
                 'v8_enable_gdbjit': 0,
                 'v8_enable_i18n_support': 0,
                 'v8_no_strict_aliasing': 1,
                 'v8_optimized_debug': 0,
                 'v8_random_seed': 0,
                 'v8_use_snapshot': 1,
                 'want_separate_host_toolset': 1}}

Realized this is pretty much a dupe of #195, but it's documented that you should be able to, when it's card coded. Also as in that issue, editing the exe.js directly makes it compile for x64

The solution from #195 unfortunately did not work for me, still giving the same error i pasted above =(

Try upgrading your Node to 6.2.0 and if you want to compile 64-bit, only pass "x64". Release is already passed by default and x86 is 32-bit

Used version 4.4.5 and that seemed to solve it! ( We want to use the lts version for our bundled issues ).
Now I just have to (continue) figuring out how to get appveyor to compile native modules.
Thanks for your help!

If it is resolved you may consider closing this issue

If I am correct the buildargs were never getting used in the code from npm. I had to use the code from github to make it work.

Correct, not sure if that's still the case.

fixed in #308


Move and zoom to portal double clicking its title.

Move and zoom to portal double clicking its title in portal detail view.
Portal click in chat doesn't center a portal and sometimes it appears under the chat (sometimes portal selection doesn't work at all for me). Another situation is when you moved the map away from the selected portal and want to see it again. Anyway if you see portal info you should have the ability to move to it simply.
Also full name of a portal is in hint now. It is useful in case it is too long and was truncated.

This is bad since double-clicking on portal title is the only way to copy/paste its full name.
If you want to recenter and zoom to a selected portal simply click on the existing "Portal link" link.

@mxxcon To copy portal name in desktop version you can just click select the title with mouse move. BTW maybe on single click it is better to add "copy to clipboard" functionality. On mobile version there is still a long click.
"Portal link" reloads the page which is not what is required espetially if you have portal list dialog opened.


Generate .csv function for new subject level results

To implement the functionality for specs outlined here:

[ ] Implement a check box to switch between the two .csv files
[ ] Modify generateCSVString to return new subject-level results (the two .csv files).
[ ] Update README.md to mention what the tool now returns based on the status of the checkbox

As part of this, we need to make sure that we return the human-readable labels to the users, not the IRI for diagnosis levels and sex. See https://github.com/neurobagel/project/issues/47
For the moment, we will use the hard-coded mapping of controlled term IRI to human-readable label that already exists in the query tool

Blocked by https://github.com/neurobagel/api/issues/112

Unlocked by neurobagel/api#125

:tada:
but also: this feels like bot work. Can it be bot work?


Parent Resolver Type Uses Return Resolver Type

Describe the bug
When using a union type in GraphQL, the output types for ResolversParentTypes will use ResolversTypes which results in a Promise wrapped value. However, in GraphQL, you can't receive a Promise<T> in the parent position. Looks to be caused by this line: https://github.com/dotansimha/graphql-code-generator/blob/f5250087ef756f02803259abeecf318cd4180e9c/packages/plugins/other/visitor-plugin-common/src/base-resolvers-visitor.ts#L344. This code is used to generate both parent resolvers types and resolvers types, but doesn't differentiate within a union.
To Reproduce
Steps to reproduce the behavior:

My GraphQL schema:

union Union = A | B

My GraphQL operations:

# Put your operations here

My codegen.yml config file:

schema: "schema.graphql"
generates:
  src/__generated__/graphql.ts:
    plugins:
      - typescript
      - typescript-resolvers

Expected behavior
It should use the parent resolvers type for a union within parent resolvers.
Additional context

Is there an ETA on this being released?

Fixed in 1.8.0 🚀


Add Web Audio Drum Machine

Hi, I added Web Audio Drum Machine

You can also checkout our Idea-Arca, the project idea bank for any languages, for open-source contributions.
Happy Contributing season!
Keep up the good work!

Thanks for the merge and the new link!


Enable integration tests to run with non-root mac_username

@paetling @jsingle @zacharym Can now provide DUSTY_INTEGRATION_TESTS_USER environment var to run with your normal mac_username. Should hopefully eliminate the pain around running integration tests locally. Only necessary change was some permission fudging.

Once this gets merged would recommend blowing away any root VM you have and running this way from now on

This PR makes me happy

Also, we directly write to your config in tests.  How is this getting by that?

good question

If we think it is better to run it this way, why do we not just default to running it with your current user and then you have to provide the environment variable to change that?

one comment but about the repos test setUp / tearDown but yeah, LGTM

@paetling The current user is always root.


Makefile: Calculate $(GOROOT), etc. relative to Makefile

Same commit as this one for alertmanager, also applies here.
When building a Debian package with debuild, which uses fakeroot, Make doesn't have $(PWD) for some reason. Apparently this is a shell-dependent thing. For that reason, the touch command gets the wrong path in $@:
curl -o .deps/go1.4.1.linux-amd64.tar.gz -L https://golang.org/dl/go1.4.1.linux-amd64.tar.gz
tar -C .deps -xzf .deps/go1.4.1.linux-amd64.tar.gz
touch /.deps/go/bin/go
touch: cannot touch ‘/.deps/go/bin/go’: No such file or directory
make[1]: *** [/.deps/go/bin/go] Error 1

This uses the initial location of the first makefile to calculate relative paths.

Thanks!


Fetching logs does not work when using OpenStack s3 object storage

What steps did you take and what happened:
When creating a backup with using OpenStack s3 object storage, backup and restore work just fine, but fetching the logs fails:
$ ark backup logs backup-test
An error occurred: request failed: <?xml version="1.0" encoding="UTF-8"?><Error>NoSuchKey<Message>The resource you requested does not exist</Message><Resource>/metakube-cluster-backup-fhgbvx65xg-ark/backup-test/backup-test-logs.gz</Resource><RequestId>2925491272881701579</RequestId></Error>

$ ark restore logs backup-test-20180830163119
An error occurred: request failed: <?xml version="1.0" encoding="UTF-8"?><Error>NoSuchKey<Message>The resource you requested does not exist</Message><Resource>/metakube-cluster-backup-fhgbvx65xg-ark/backup-test/restore-backup-test-20180830163119-logs.gz</Resource><RequestId>1135366011835801288</RequestId></Error>

The files exist though and also have the correct valid content:
$ s3cmd la --recursive
2018-08-30 14:48         0   s3://metakube-cluster-backup-fhgbvx65xg-ark/backup-test/
2018-08-30 14:13      2446   s3://metakube-cluster-backup-fhgbvx65xg-ark/backup-test/backup-test-logs.gz
2018-08-30 14:13      5907   s3://metakube-cluster-backup-fhgbvx65xg-ark/backup-test/backup-test.tar.gz
2018-08-30 14:31       647   s3://metakube-cluster-backup-fhgbvx65xg-ark/backup-test/restore-backup-test-20180830163119-logs.gz
2018-08-30 14:31        82   s3://metakube-cluster-backup-fhgbvx65xg-ark/backup-test/restore-backup-test-20180830163119-results.gz

$ s3cmd get s3://metakube-cluster-backup-fhgbvx65xg-ark/backup-test/restore-backup-test-20180830163119-logs.gz
download: 's3://metakube-cluster-backup-fhgbvx65xg-ark/backup-test/restore-backup-test-20180830163119-logs.gz' -> './restore-backup-test-20180830163119-logs.gz'  [1 of 1]
 647 of 647   100% in    0s   110.17 kB/s  done

$ gunzip restore-backup-test-20180830163119-logs.gz

$ cat restore-backup-test-20180830163119-logs
time="2018-08-30T14:31:21Z" level=info msg="Not including resource" groupResource=events logSource="pkg/restore/restore.go:124"
...

What did you expect to happen:
Logs are displayed correctly
The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)

kubectl logs deployment/ark -n heptio-ark
No errors.
ark backup describe <backupname> or kubectl get backup/<backupname> -n heptio-ark -o yaml
No errors.
ark backup logs <backupname>
Doesn't work, see above. No errors when manually downloading and viewing them.
ark restore describe <restorename> or kubectl get restore/<restorename> -n heptio-ark -o yaml
No errors.
ark restore logs <restorename>
Doesn't work, see above.  No errors when manually downloading and viewing them.

Environment:

Ark version (use ark version):  0.9.3
Kubernetes version (use kubectl version): 1.11.2 on server and client
Cloud provider or hardware configuration: OpenStack
OS (e.g. from /etc/os-release): Ubuntu Xenial

I wonder if the leading / in e.g. /metakube-cluster-backup-fhgbvx65xg-ark/backup-test/restore-backup-test-20180830163119-logs.gz is causing the problem.
@bashofmann can you run aws s3 presign s3://metakube-cluster-backup-fhgbvx65xg-ark/backup-test/restore-backup-test-20180830163119-logs.gz and share the returned url?

The result is:
$ aws --endpoint-url https://s3.cbk.cloud.syseleven.net s3 presign s3://metakube-cluster-backup-fhgbvx65xg-ark/backup-test/restore-backup-test-20180830163119-logs.gz
https://s3.cbk.cloud.syseleven.net/metakube-cluster-backup-fhgbvx65xg-ark/backup-test/restore-backup-test-20180830163119-logs.gz?AWSAccessKeyId=xxxx&Signature=xxxx%3D&Expires=1535647777

For reference our ark config is:
apiVersion: ark.heptio.com/v1
backupStorageProvider:
  bucket: metakube-cluster-backup-fhgbvx65xg-ark
  config:
    region: cbk
    s3ForcePathStyle: "true"
    s3Url: https://s3.cbk.cloud.syseleven.net
  name: aws
  resticLocation: metakube-cluster-backup-fhgbvx65xg-restic
backupSyncPeriod: 1m
gcSyncPeriod: 1m
kind: Config

Hmm, I'm not seeing anything in the code that would put a leading / at a quick glance. What OpenStack version are you running?

I debugged this and the path of the download URL the ark server returns to the CLI is correct:
https://s3.cbk.cloud.syseleven.net/metakube-cluster-backup-fhgbvx65xg-ark/backup-test/restore-backup-test-20180830163119-logs.gz

But the query parameters are completely different to what aws s3 presign returns:
?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=xxx%2F20180830%2Fcbk%2Fs3%2Faws4_request&X-Amz-Date=20180830T162545Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=yyy

If I replace these query parameters with AWSAccessKeyId, Signature and Expires, the URL works.

The same problem is there when trying to download a backup with the cli, by the way.

What OpenStack version are you running?

We are running OpenStack Mitaka and use Quobyte 2.5 for S3 Storage

So s3 is directly provided by Quotbyte? OpenStack isn't really relevant/involved?

Sorry for the late reply, I checked with our OpenStack team and yes S3 is directly provided by Quobyte. It seems that Quobyte only supports the V2 Signer API, but ark uses V4.
Would it be possible to make the version configurable? For example like this:
func (o *objectStore) CreateSignedURL(bucket, key string, ttl time.Duration) (string, error) {
	req, _ := o.s3.GetObjectRequest(&s3.GetObjectInput{
		Bucket: aws.String(bucket),
		Key:    aws.String(key),
	})

	if o.signatureVersion == "2" {
		req.Handlers.Sign.Remove(v4.SignRequestHandler)
		req.Handlers.Sign.PushBackNamed(v2.SignRequestHandler)
	}

	return req.Presign(ttl)
}

@bashofmann we can look into this. I checked the aws sdk, and the v2 SignRequestHandler is in a private package. We can import it and use it, but Amazon's guidance is that anything in there is for the SDK's internal use and is subject to breaking changes.
We would need to define a new key for our aws ObjectStore's config to specify signature version. Would you be interested in trying to put together a PR for this?

@bashofmann the aws go sdk does have a v2 signer, if that's what you need (with the caveat from the repo that it's a private api and may change/go away/etc).
You also could do a separate quobyte plugin if you wanted to. We have #193 as a possible way to make it easier to have common AWS behavior with differences here and there (e.g. quobyte signatures), but we haven't implemented it yet.

The v2 signer unfortunately does not work, I already tried it. And from the python code there are some differences between v1 and v2.

Ah, that's unfortunate. I'll talk with the team about whether or not we want to own a v1 signing algorithm in the Ark code base and get back to you.

FYI: I with the V1 algorithm downloading logs and backups works correctly with Quobyte S3

cc @skriss

@ncdc @skriss Did you already have time to discuss if we should contribute this as a PR to ark or write a plugin for this?

We haven't had a chance to chat, but I'd be ok with a PR in the short term, and #193 long term. WDYT @skriss?