Offline/Online LP for CIM scenario 📚

Description
Add Offline and Online Linear programming agent for CIM scenario

Link: paper 1 paper 2
Screenshot:

Hi @riccardopoiani , thank you for your attention to our work. Currently, we don't have the plan to transfer all these methods used in the paper to MARO platform. If you have some questions about the implementation, we can have a discussion. Thanks again!

Hi @Jinyu-W, how about the methods proposed in these papers. Does the current MARO contains the implementation of those? In "Cooperative Policy Learning with Pre-trained Heterogeneous Observation Representations paper, it is written that source code is attached in 'code' folder but I couldn't find it. Can you help me, please. Thank you very much :)

Hi @Jinyu-W, how about the methods proposed in these papers. Does the current MARO contains the implementation of those? In "Cooperative Policy Learning with Pre-trained Heterogeneous Observation Representations paper, it is written that source code is attached in 'code' folder but I couldn't find it. Can you help me, please. Thank you very much :)

@wesley-stone

Hi @Jinyu-W, thank you for the response. I would be grateful to @wesley-stone for sharing code and related info. Many thanks again! :)

Hi @Jinyu-W thanks for your response and availability.
I am trying to replicate results for Offline and Online LP in different configurations.
Currently, I am able to achieve 0% and 1% shortage in toy domains with Offline and Online LP respectively.
However, when I try to run the program on global trade configurations, I obtain 4% with Offline and 33% with the Online one.
So, first of all, I have seen that there is an open pull request to adjust the topology for CIM and I was wondering if this was the cause. However, no issue describes the change made there. So, I would start from here, and then maybe we can build a discussion on how to implement correctly the method.
Also, have you got any method that is able to reach high performances with the current version of the simulator?

Hi @Jinyu-W thanks for your response and availability.
I am trying to replicate results for Offline and Online LP in different configurations.
Currently, I am able to achieve 0% and 1% shortage in toy domains with Offline and Online LP respectively.
However, when I try to run the program on global trade configurations, I obtain 4% with Offline and 33% with the Online one (which are significantly higher than the ones in the papers).
So, first of all, I have seen that there is an open pull request to adjust the topology for CIM and I was wondering if this was the cause. However, no issue describes the change made there. So, I would start from here, and then maybe we can build a discussion on how to implement correctly the method.
Also, have you got any method that is able to reach high performances with the current version of the simulator? (any sort of method)

Hi @riccardopoiani, I could share more information about the LP methods we used before.

The Online LP formulated based on the demand(orders) predicted with a simple MA model.
Besides the reward for the order fulfillment, we also added the reward for safety inventory in the objective function. For example, for each tick, if we have empty containers remained after fulfilling all the orders of this tick, an additional reward would be given: safety_inventory_reward_factor * min(port capacity * safety inventory upper ratio, #remaining empty containers).
The main difference between the offline LP and the online LP is that the actual demand(orders) is given instead of the prediction.

We may open source a version of Online LP in the future, but it will not happen immediately, it will take some time.

More information about the paper, the author @wesley-stone  will have more say. :)

@Jinyu-W thanks for you feedback. I will try for sure to integrate the safety reward factor.
For what concerns the Online LP, are the methods in paper 1 and paper 2 differents?
In paper 1 they state that "the exact future order information to replace the forecasted future demand in the LP model so as to eliminate the effects of external factors leading to uncertain forecasts" (section 5.3).
Indeed, when I talked to the first author,  he said that the main difference between Online and Offline was that the plan (in the offline one) was not fed to the simulator, and so, "illegal" actions were not corrected to make them feasible.
Moreover, it seems to me that the crucial point to have good performances (when feeding the plan to the simulator) is that you need estimates that are as accurate as possible (otherwise the plan will soon diverge from reality).
The crucial point in this step is that you want to estimate the future laden of containers at your best (from the laden you can recover estimates of supplies and vessels capacity for empty containers) . How do you do that?  Is there anything specific I should pay attention to?

Also, @Ahmedest61 I have seen something similar to what you are looking for in temp_gnn branch, under the folder examples/cim/ac_gnn, but I have not tried it.

Ah, thank you so much @riccardopoiani. I'll have a a look. Just wondering, is there any difference in terms 'gnn-ac' (or any other methodology) used 'temp_gnn' and 'maro-0.2_cim_topology_adjustment' branches? If not, I assume 'maro-0.2_cim_topology_adjustment' is the right, more stable one? Thank you!

No idea on this @Ahmedest61. I am currently working on master. Please, let me know if the code I pointed you to is what you were looking for and if you are able to achieve good performances (at least ~90% container container satisfaction on global configs).

Yup, I think this branch "temp_gnn" is similar to what paper 2 proposed but again you're right about PreLAC part!

@Jinyu-W thanks for you feedback. I will try for sure to integrate the safety reward factor.
For what concerns the Online LP, are the methods in paper 1 and paper 2 differents? Indeed I have some conflicting information about Operation Research baselines.
In paper 1 they state that "the exact future order information to replace the forecasted future demand in the LP model so as to eliminate the effects of external factors leading to uncertain forecasts" (section 5.3 - Online LP).
Indeed, when I talked to the first author, he said that the main difference between Online and Offline was that the plan (in the offline one) was not fed to the simulator, and so, "illegal" actions were not corrected to make them feasible.
Moreover, it seems to me that the crucial point to have good performances (when feeding the plan to the simulator) is that you need estimates that are as accurate as possible (otherwise the plan will soon diverge from reality).
The crucial point in this step is that you want to estimate the future laden of containers at your best (from the laden you can recover estimates of supplies and vessels capacity for empty containers) . How do you do that? Is there anything specific I should pay attention to?

Hi, as far as I know, the LP methods used in paper 1 and paper 2 are not the same. The one used in paper 2 is more closer to the formulation I have done before. So I could say more about it. As you mentioned above, the accuracy of the prediction can affect the final performance a lot, to avoid the prediction error being magnified over time. I have added two more "time window" to the LP solution formulation:

One for the problem formulation/solution. Assume we use a time window for 200 ticks, no matter how long the simulation is, each time we only look forward to 200 ticks (also the prediction), and to get the solution for only the future 200 ticks.
One for the solution applying. The apply window size <= the formulation window size. Assume we use an apply window size of 20 ticks. Then the formulation and applying trace would be: formulating and solving at tick 0, applying the solution..., formulating and solving at tick (0 + 20), applying the solution..., formulating and solving at tick (0 + 20 + 20), .... Of course, as you may know, currently the simulation is triggered by the events. If you trigger the problem formulation by the event, the actual ticks for the formulation may be different from the single example above. Like: tick 7 - event coming, formulating, applying ... ; tick 14 - event coming, applying, ...; tick 21 - event coming, formulating, applying, ... ; tick 28 - event coming, formulating, applying, ...

Wish these may help you.

Hi, as far as I know, the LP methods used in paper 1 and paper 2 are not the same. The one used in paper 2 is more closer to the formulation I have done before. So I could say more about it. As you mentioned above, the accuracy of the prediction can affect the final performance a lot, to avoid the prediction error being magnified over time. I have added two more "time window" to the LP solution formulation:
1. One for the problem formulation/solution. Assume we use a time window for 200 ticks, no matter how long the simulation is, each time we only look forward to 200 ticks (also the prediction), and to get the solution for only the future 200 ticks.

2. One for the solution applying. The apply window size <= the formulation window size. Assume we use an apply window size of 20 ticks. Then the formulation and applying trace would be: formulating and solving at tick 0, applying the solution..., formulating and solving at tick (0 + 20), applying the solution..., formulating and solving at tick (0 + 20 + 20), .... Of course, as you may know, currently the simulation is triggered by the events. If you trigger the problem formulation by the event, the actual ticks for the formulation may be different from the single example above. Like: **tick 7** - event coming, **formulating,** applying ... ; tick 14 - event coming, applying, ...; tick 21 - event coming, formulating, applying, ... ; **tick 28** - event coming, **formulating**, applying, ...

Wish these may help you.

Thanks for the hints, but unfortunately, I am already doing both things. I tried many combinations of these hyperparameters but they not seem to solve my issue. It seems to be something that is more related to the estimation of the laden containers.
Currently, to estimate laden, I first compute the exact future demands in advance. Then, using CimDataCollection I pre-simulate the ordered arrivals of vessels. When there is a vessel arrival, I do the following:

Take the estimated laden that vessel is carrying for port_idx and discharge it (this will lead to a supply for port_idx at arrive_tick)
Iterate over the next future arrivals of vessel and load laden according to the cumulated unsatisfied demand that port_idx has for each of the vessel destination
Compute the remaining capacity to estimate the capacity for empty containers

Am I missing something relevant here? Is this the way in which you estimate laden (and thus the supply and the vessel capacity for empty containers)?

Hi @riccardopoiani, I could share more information about the LP methods we used before.
* The Online LP formulated based on the demand(orders) predicted with a simple MA model.

* Besides the reward for the order fulfillment, we also added the reward for safety inventory in the objective function. For example, for each tick, if we have empty containers remained after fulfilling all the orders of this tick, an additional reward would be given: safety_inventory_reward_factor * min(port capacity * safety inventory upper ratio, #remaining empty containers).

* The main difference between the offline LP and the online LP is that the actual demand(orders) is given instead of the prediction.

We may open source a version of Online LP in the future, but it will not happen immediately, it will take some time.

I also noticed here that you were mentioning port_capacity. However, I was not able to find any usage of this attribute in the simulator. Is this a bug or I wasn't able to find it? It does not seems like port capacity is checked when actions are taken.
Searching in files with .capacity does not report any usage of that attribute to me.

Hi @riccardopoiani, I could share more information about the LP methods we used before.
* The Online LP formulated based on the demand(orders) predicted with a simple MA model.

* Besides the reward for the order fulfillment, we also added the reward for safety inventory in the objective function. For example, for each tick, if we have empty containers remained after fulfilling all the orders of this tick, an additional reward would be given: safety_inventory_reward_factor * min(port capacity * safety inventory upper ratio, #remaining empty containers).

* The main difference between the offline LP and the online LP is that the actual demand(orders) is given instead of the prediction.

We may open source a version of Online LP in the future, but it will not happen immediately, it will take some time.

I also noticed here that you were mentioning port_capacity. However, I was not able to find any usage of this attribute in the simulator. Is this a bug or I wasn't able to find it? It does not seems like port capacity is checked when actions are taken.
Searching in files with .capacity does not report any usage of that attribute to me.

According to our experience before, the port capacity will not be the limitation for the valid action scope, and we also set a big enough capacity for the ports in those topologies. So you the absence of the checking of the port capacity wouldn't make any difference. But I agree that it's better to add it back.

Hello @Ahmedest61, the code' folder was the attached file when we submitted the paper to AAMAS. Currently our method has been implemented in the branch v0.2_cim_topology_adjustment' https://github.com/microsoft/maro/tree/v0.2_cim_topology_adjustment/examples/cim/gnn :).

Yup, I think this branch "temp_gnn" is similar to what paper 2 proposed but again you're right about PreLAC part!

Yes, you are right :)

Hi, as far as I know, the LP methods used in paper 1 and paper 2 are not the same. The one used in paper 2 is more closer to the formulation I have done before. So I could say more about it. As you mentioned above, the accuracy of the prediction can affect the final performance a lot, to avoid the prediction error being magnified over time. I have added two more "time window" to the LP solution formulation:
1. One for the problem formulation/solution. Assume we use a time window for 200 ticks, no matter how long the simulation is, each time we only look forward to 200 ticks (also the prediction), and to get the solution for only the future 200 ticks.

2. One for the solution applying. The apply window size <= the formulation window size. Assume we use an apply window size of 20 ticks. Then the formulation and applying trace would be: formulating and solving at tick 0, applying the solution..., formulating and solving at tick (0 + 20), applying the solution..., formulating and solving at tick (0 + 20 + 20), .... Of course, as you may know, currently the simulation is triggered by the events. If you trigger the problem formulation by the event, the actual ticks for the formulation may be different from the single example above. Like: **tick 7** - event coming, **formulating,** applying ... ; tick 14 - event coming, applying, ...; tick 21 - event coming, formulating, applying, ... ; **tick 28** - event coming, **formulating**, applying, ...

Wish these may help you.

Thanks for the hints, but unfortunately, I am already doing both things. I tried many combinations of these hyperparameters but they not seem to solve my issue. It seems to be something that is more related to the estimation of the laden containers.
Currently, to estimate laden, I first compute the exact future demands in advance. Then, using CimDataCollection I pre-simulate the ordered arrivals of vessels. When there is a vessel arrival, I do the following:
1. Take the estimated laden that `vessel` is carrying for `port_idx` and discharge it (this will lead to a supply for `port_idx` at `arrive_tick`)

2. Iterate over the next future arrivals of `vessel` and load laden according to the cumulated unsatisfied demand that `port_idx` has for each of the vessel destination

3. Compute the remaining capacity to estimate the capacity for empty containers

Am I missing something relevant here? Is this the way in which you estimate laden (and thus the supply and the vessel capacity for empty containers)?
Moreover, what are the most relevant changes between paper 1 and paper 2 that you are aware of?

Any answer for this?

Hi @wesley-stone, could any of you provide the environment configuration that you used in paper 2?
With master/global_trade_cfg I have results of 68% fulfillment, while with these configs I am able to reach 98-100% fulfillment (same LP method). Thus, I was wondering what cfg you used in the paper so that I can actually compare results.

Hi, @wesley-stone @Jinyu-W any update on the previous post?
It seems to me somehow relevant to provide good configurations within the environment. Also because in these configs no repositioning performs very good while with master branch configurations it seems to me that the optimal solution is very far from 100%.

Hi, as far as I know, the LP methods used in paper 1 and paper 2 are not the same. The one used in paper 2 is more closer to the formulation I have done before. So I could say more about it. As you mentioned above, the accuracy of the prediction can affect the final performance a lot, to avoid the prediction error being magnified over time. I have added two more "time window" to the LP solution formulation:
1. One for the problem formulation/solution. Assume we use a time window for 200 ticks, no matter how long the simulation is, each time we only look forward to 200 ticks (also the prediction), and to get the solution for only the future 200 ticks.

2. One for the solution applying. The apply window size <= the formulation window size. Assume we use an apply window size of 20 ticks. Then the formulation and applying trace would be: formulating and solving at tick 0, applying the solution..., formulating and solving at tick (0 + 20), applying the solution..., formulating and solving at tick (0 + 20 + 20), .... Of course, as you may know, currently the simulation is triggered by the events. If you trigger the problem formulation by the event, the actual ticks for the formulation may be different from the single example above. Like: **tick 7** - event coming, **formulating,** applying ... ; tick 14 - event coming, applying, ...; tick 21 - event coming, formulating, applying, ... ; **tick 28** - event coming, **formulating**, applying, ...

Wish these may help you.

Thanks for the hints, but unfortunately, I am already doing both things. I tried many combinations of these hyperparameters but they not seem to solve my issue. It seems to be something that is more related to the estimation of the laden containers.
Currently, to estimate laden, I first compute the exact future demands in advance. Then, using CimDataCollection I pre-simulate the ordered arrivals of vessels. When there is a vessel arrival, I do the following:
1. Take the estimated laden that `vessel` is carrying for `port_idx` and discharge it (this will lead to a supply for `port_idx` at `arrive_tick`)

2. Iterate over the next future arrivals of `vessel` and load laden according to the cumulated unsatisfied demand that `port_idx` has for each of the vessel destination

3. Compute the remaining capacity to estimate the capacity for empty containers

Am I missing something relevant here? Is this the way in which you estimate laden (and thus the supply and the vessel capacity for empty containers)?
Moreover, what are the most relevant changes between paper 1 and paper 2 that you are aware of?

Any answer for this?

@riccardopoiani  Sorry for the late reply. Originally I want to reproduce the online LP for CIM and share it with you, but I was busy with some other works :( As for the points in your problem formulation:

For the vessel arrival, I think it is OK to read from the Data Collection.
For the demand of each port, we use the predicted value in the Online solution. And for the ones consumed by the historical demand, we'll add it into the future supply of the specific ports accordingly.
For the future supply of each port, we also use the predicted value.
Also, as you already formulated, both the demand and supply would be used to calculate the future inventory (and remaining space) for both ports and vessels.

You can refer to this semifinished example (for reference only) to get more details. But currently, limited by the predicted value input, still cannot get the optimal solution.
As for the change between paper 1 and paper 2, sorry but I don't have any idea about it. As for the details of paper 2, sorry but I'm not the author and I don't know the details either.

Thank you very much @Jinyu-W.
I just had a few tests on the environment and this somehow confirmed me what I was thinking.
The summary is the following: I believe that the current global trade configurations are "too hard" to achieve reasonably performances. Currently, all the methods that I have implemented and tested can't get beyond 70% order fulfillment (even in the simplest configuration, that is global_trade.22p_l0.0.
Instead, what happens, is that just with a slight decrease in container_usage_proportion, in particular, I decreased its value by a factor of 0.008 or 0.007) depending on the configuration. So, for instance, in global_trade.22p_l0.0:
container_usage_proportion:
  period: 112
  sample_nodes:
  - - 0
    - 0.02
  - - 111
    - 0.02
  sample_noise: 0

is substituted with
container_usage_proportion:
  period: 112
  sample_nodes:
  - - 0
    - 0.012
  - - 111
    - 0.012
  sample_noise: 0

This actually helped in reaching very good performances.
Another think that I tested, is to increase the vessel speed (3x), actually helped in reaching the same high performances.
Instead,  increasing the capacities of vessels and ports to a very large number didn't help in increasing the performances of these methods (they were stuck below 70%).
Of course, I cannot know if there are bugs in my implementation, but I'm sure at 99% that the problem is in the plausible unfeasibility of the environment.
The fact that this happened even with your implementation, is even a stronger signal.
Indeed, with the global configuration that I have been proposing, your method achieve around 90% in global_trade.22p_l0.0 (that has constant order distributions). While, with the current configuration it only gets around 63%.
I would like that you consider the possibility that actually the current configurations are "too hard" to be solved with good performances. In this sense I see two options: recover in some way the configurations used in the paper, or manually decrease that hardness as I did.

Firstly, I want to express my heartfelt thanks for your careful and considerable verification @riccardopoiani. There are 2 things I want to clarify:

this semifinished example and the one used in paper 2 share a similar problem formulation, but it is not certain whether the one used in paper 2 is exactly the same as the example.
My personal view: I don't think it is reasonable to adjust the topology according to the performance. It should be reversed. Determine the topology based on the data distribution or some reliable analysis, then tune the algorithm (no matter ML-based/OR-based/heuristic/...) to get better performance. So, if there are some unreasonable settings in the topologies, we will be willing to update them.
Thanks again!

Thank you very much again @Jinyu-W.
You are right that the configurations should not be tuned to reach good performances. It make perfectly sense.
However, I would expect at least the repository to contain information on the best results that have been achieved by some method of yours on that environment.
I think that reporting performances of no-repositioning and random actions (as in maro's doc) is probably not enough.
For instance, when dealing with gym environment, you have information like "CartPole-v0 defines "solving" as getting average reward of 195.0 over 100 consecutive trials" (as written here).
This let understand who is experimenting with the repo for research purpose, how sub-optimal their algorithm is w.r.t. different configurations. Not having such feature, complicate significantly the job of understanding whether a new good method has been designed or not.
If you still consider this not be enough, I will understand.
Thanks again for your replies :)

Thank you very much again @Jinyu-W.
You are right that the configurations should not be tuned to reach good performances. It make perfectly sense.
However, I would expect at least the repository to contain information on the best results that have been achieved by some method of yours on that environment.
I think that reporting performances of no-repositioning and random actions (as in maro's doc) is probably not enough.
For instance, when dealing with gym environment, you have information like "CartPole-v0 defines "solving" as getting average reward of 195.0 over 100 consecutive trials" (as written here).
This let understand who is experimenting with the repo for research purpose, how sub-optimal their algorithm is w.r.t. different configurations. Not having such feature, complicate significantly the job of understanding whether a new good method has been designed or not.
If you still consider this not be enough, I will understand.
Thanks again for your replies :)
EDIT: Let me rephrase it a bit: the main point of having feasible environments (or the ones tested in the papers) is because you have already something to compare yourself to. E.g.: with feasible environments, if your method is good you should have performances close to 100%. If you have the results in the paper, you should have performances close to the ones in the paper.
With current envs, instead, one would expect to obtain performances close to the ones in the paper (since it is a global trade configuration), while this is probably not possible. This should indeed at least be specified in the documentation (so that other people can actually avoid the same error I did).
Moreover, as default configuration I would expect to have something that is in some way "learnable". It does not make much sense to me to have an environment in which basically all the methods performs approx. the same.

I am just getting into this conversation now, and I hope to provide some useful information. @Jinyu-W please correct me if I am wrong. Concerning comparison with the results in the papers, I think one of the problems comes from using different cost functions, not only among the papers, but also between the papers and the current code. This is true for both the LP model and the RL based model (if you are using the default reward function in main, this is way simpler than the one in the papers).
@riccardopoiani maybe I missed this in the previous messages above, have you tried to use the Cost function in the paper for the LP solver ? I did not, and it is in my TODO list, just curious. Same thing apply for the RL models. I think that changing environment parameters is not the way to go, although I totally agree that it would be good to have a benchmark to use as a reference.
Last question/observation: are you sure that the envs are not feasible? Feasibility of a problem (as a whole) is often independent from the solver we use. What I mean is: say we have a simple supply/demand problem and we need to satisfy all demand as a constraint; if supply << demand, we will never be able to find a feasible solution. Ok I know this is embarrassingly simplistic.
As constraints becomes tighter, the feasible space shrinks and the solver will have problems to find a feasible solution. So here a better solver, or implementing some constraint relaxation may help.
Bottom line, I was wondering if there is a way to have a bound on the feasibility of the problem. The provided sample problems are classified for increasing difficulty, based on the order distribution, the noise etc. Despite how hard is to find a good solution, does the problem at least admits a feasible solution?
This could help us understand if a certain result could be, at lest in principle, improved by using another method.

I am just getting into this conversation now, and I hope to provide some useful information. @Jinyu-W please correct me if I am wrong. Concerning comparison with the results in the papers, I think one of the problems comes from using different cost functions, not only among the papers, but also between the papers and the current code. This is true for both the LP model and the RL based model (if you are using the default reward function in main, this is way simpler than the one in the papers).
@riccardopoiani maybe I missed this in the previous messages above, have you tried to use the Cost function in the paper for the LP solver ? I did not, and it is in my TODO list, just curious. Same thing apply for the RL models. I think that changing environment parameters is not the way to go, although I totally agree that it would be good to have a benchmark to use as a reference.

Hello @WessZumino, both papers report performances in terms of container shortage (even if they use a different reward function and so on), the final metric is always the same. And that is the kind of performances that I am using as well, even if currently my objective function/reward functions are functions of the environments that are a bit more more complex. (i.e., I am not using -shortage in RL to achieve good results). The point is that the final goal stay the same: in other words, I am not taking into account goals that contrast this objective, such as "move as less container as possible in order to minimize the transfer cost while minimizing the shortage". In other words, even if different reward function/objective function are used, the final results are still comparable since the final goal is the one of minimizing the container shortage.
For what concern your question, I have tried many things and many reward/objective function so far in my personal implementation of the LP baseline. I have not for what concern the LP method provided in maro.

Last question/observation: are you sure that the envs are not feasible? Feasibility of a problem (as a whole) is often independent from the solver we use. What I mean is: say we have a simple supply/demand problem and we need to satisfy all demand as a constraint; if supply << demand, we will never be able to find a feasible solution. Ok I know this is embarrassingly simplistic.
As constraints becomes tighter, the feasible space shrinks and the solver will have problems to find a feasible solution. So here a better solver, or implementing some constraint relaxation may help.
Bottom line, I was wondering if there is a way to have a bound on the feasibility of the problem. The provided sample problems are classified for increasing difficulty, based on the order distribution, the noise etc. Despite how hard is to find a good solution, does the problem at least admits a feasible solution?
This could help us understand if a certain result could be, at lest in principle, improved by using another method.

First of all, I apologize for the confusion. With unfeasible envs, I mean that there is no way of obtaining 0 shortage.
This is different from the method being unable to return any solution. All the implementations I have tested are able to compute a feasible plan, the point is that the performance of the plan are not what one may expect after reading the papers.
Of course I am not sure since I am not sure that my implementation is bug free.
However, the LP method mentioned in paper 1 is based on the assumption that all demands can be satisfied (i.e., minimizing the shortage, you assume that the optimal objective function is 0, or close to 0).
If this is the case (the environment is "feasible"), the method performs as an oracle (indeed, in that algorithm all the unknown information such as demands, arrivals and so on are known in advance, at planning time (this info comes from the first author of the paper)). I am able to obtain performances > 96%  in all configurations that satisfy this constraints. As soon as this requirement is not satisfied performances start to decrease. The more this constraint is not satisfied, the more the drop in performance.

Hello @WessZumino, both papers report performances in terms of container shortage (even if they use a different reward function and so on), the final metric is always the same. And that is the kind of performances that I am using as well, even if currently my objective function/reward functions are functions of the environments that are a bit more more complex. (i.e., I am not using -shortage in RL to achieve good results). The point is that the final goal stay the same: in other words, I am not taking into account goals that contrast this objective, such as "move as less container as possible in order to minimize the transfer cost while minimizing the shortage". In other words, even if different reward function/objective function are used, the final results are still comparable since the final goal is the one of minimizing the container shortage.

Hi @riccardopoiani thanks for the clarification, it is very helpful. Concerning your point, I totally agree that the final metric is the same and therefore it is meaningful to compare to it.
The part that confuses me is the following, and maybe it is just terminology: if my goal is to compare my result to the results reported in the papers, changing objective/reward doesn't mean I am changing the underlying model ? And here by model I mean what is represented by the objective function. If I think about a physical model, the model is entirely defined by an Hamiltonian H (that is the equivalent of a cost function to minimize); say I am interested in evaluating a certain metric related to H. If I add new terms to H and evaluate the same metric M, can I say I am comparing the same model ? Should I even expect the same result for M coming from different Hs ? In physics the answer is no, but maybe it is fine within this context, I am not sure I understand.
Now lets say everything is the same as the paper: same objective, same training strategy etc. Can I reproduce the results? Maybe you have already done this and I did not understand, so I am sorry in advance if I said something trivial.

First of all, I apologize for the confusion. With unfeasible envs, I mean that there is no way of obtaining 0 shortage.
This is different from the method being unable to return any solution. All the implementations I have tested are able to compute a feasible plan, the point is that the performance of the plan are not what one may expect after reading the papers.
Of course I am not sure of my claims since I am not sure that my implementation is bug free.
However, the LP method mentioned in paper 1 is based on the assumption that all demands can be satisfied (i.e., minimizing the shortage, you assume that the optimal objective function is 0, or close to 0).
If this is the case (the environment is "feasible"), the method performs as an oracle (indeed, in that algorithm all the unknown information such as demands, arrivals and so on are known in advance, at planning time (this info comes from the first author of the paper)). I am able to obtain performances > 96% in all configurations that satisfy this constraints. As soon as this requirement is not satisfied performances start to decrease. The more this constraint is not satisfied, the more the drop in performance.
This is basically, the main reason why I started to think that it is not possible to reach good performances in current global envs (the method achieves around 60-65% depending on the different settings)
EDIT: This is "confirmed" in principle also by the fact that if you increase the vessel capacity to infinity, it does not affect the solution that you can find at all (even knowing all the info in advance).

Thanks, this is really useful information !

Hi @riccardopoiani thanks for the clarification, it is very helpful. Concerning your point, I totally agree that the final metric is the same and therefore it is meaningful to compare to it.
The part that confuses me is the following, and maybe it is just terminology: if my goal is to compare my result to the results reported in the papers, changing objective/reward doesn't mean I am changing the underlying model ? And here by model I mean what is represented by the objective function. If I think about a physical model, the model is entirely defined by an Hamiltonian H (that is the equivalent of a cost function to minimize); say I am interested in evaluating a certain metric related to H. If I add new terms to H and evaluate the same metric M, can I say I am comparing the same model ? Should I even expect the same result for M coming from different Hs ? In physics the answer is no, but maybe it is fine within this context, I am not sure I understand.

Yes @WessZumino, sorry for the misunderstanding; I thought you were mentioning more complex objectives as other papers on ECRs do. You are right that even in this case you would obtain different plans, and therefore different evaluations for M.
Just as a final clarification on this; terms that you add on H are included only to handle difference that you might encounter when you execute your plan in the simulator. For instance, this is the case for the safety inventory bonus: you want to avoid to plan in a way that satisfy the demands in a "too tight" way. Adding that bonus you incentivize the LP method to create a stock of containers.  In this context, my experience so far is the following: bonus that you can add to your objective/reward function can help, but the improvement is not incredibly significant. For instance, depending on the configuration, you can go gain a 2% in the fulfillment. In other words, you can expect slightly better results.

Now let us say everything is the same as in the paper: same objective, same training strategy etc. Can I reproduce the results? Maybe you have already done this and I did not understand, so I am sorry in advance if I said something trivial.

I've been trying so far. The problem is that all the global env cfgs are way different from the ones of both papers.
This is clear also from the performance of the no-repositioning policy, that can obtain from 59-63% on the current configurations vs the 81.95 reported in the paper.
With the tweaked configurations that I was mentioning above, the performance of the no-repositioning policy is around 82-83%, and the OnlineLP of paper 1 (that is different from paper 2 and the one in the maro branch) obtains 96%.
Of course, I have no idea is the configuration is the same, but at least performances seems to be more "comparable".


Updating version of Jackson because of an error

Using 2.7.0 I get an error between you library and java library for bots where is the version of Jackson 2.8.7:
exception in thread "main" java.lang.NoSuchMethodError: com.fasterxml.jackson.core.JsonGenerator.writeStartObject(Ljava/lang/Object;)V
but when I specify explicitly in my own POM file version higher than 2.8.0, all is ok.

Coverage remained the same at 97.727% when pulling f4eb42777458b0ca0a9bb8a977972843cfcac2ca on invzbl3:patch-1 into b8cc0c60b8d804cfd7f3369689576b9a93ea1e6f on kcthota:master.


Loosen static shapes on deferred calls to allow cellular batching

After identifying good targets for cellular batching (#41) we'll need to ensure we can actually batch. Though coalescing is possible even if batching is not and often still provides throughput benefits the real wins come from increasing arithmetic density of the GEMVs. We should be able to detect which shape dimensions we can make partial for a given deferred call body and do so.

This will likely shake out from other work related to semantically deduping executables and reducing compiled binary sizes.


How do I use the commands in HERO5-Commands.md?

Problem:
Title says it.
Details:

GoPro Camera(s): Hero5 Black
Firmware Version: 02.51
Steps to reproduce:
Happens every time? [Y/N]:

Use the URLs with a HTTP library of your choice (eg: curl, aria2, urllib, requests...)


Koa POST requests hang using http2

Sometimes making a POST or PUT request to my Koa based server hangs. No error, nothing.
At the subsequent POST request both calls are executed, first the new call, then the old one completes.
The server uses HTTP2 and it is written in Typescript. Unfortunately a small example to reproduce the problem does not exhibit any hang.
I tried to look at similar problems on the web. They speak of bodyparser problems for malformed client requests, but here the client sends a simple JSON encoded string. Also moving http2 to spdy does not work, spdy crashes with Node version 12.
My middleware stack is as follow:
this.app = new Koa();
this.app
	.use(compress({threshold: 1024 * 64}))
	.use(favicon(path.join(__dirname, "http", "favicon.ico")))
	.use(serve(path.join(__dirname, "http"), {maxage: ms("1y")}))
	.use(this.noop)
	.use(bodyParser({
		jsonLimit: "10mb",
		formLimit: "10mb",
		textLimit: "10mb",
		onerror: function(error, ctx) {
				ctx.throw(422, `Body parse error: ${error}`);
		}
	}))
	.use(routes.startRoutes(this.resources));

You see the this.noop one? It is a do nothing middleware:
private async noop(ctx: Koa.Context, next: Koa.Next): Promise<void> {
	// console.log("Test start");
	await next();
	// console.log("Test end");
}

If I uncomment the console.log calls, the problem disappears. So seems a race or timing problem somewhere.
Also, if I switch to HTTP, the problem does not appear. So at least I have a Plan B to continue development.
Working on Windows 10 64bits with Node 12.16.1 and the koa packages:
"koa": "^2.11.0",
"koa-bodyparser": "^4.3.0",
"koa-compress": "^4.0.0",
"koa-favicon": "^2.1.0",
"koa-logger": "^3.2.1",
"koa-router": "^8.0.8",
"koa-static": "^5.0.0",

Any suggestion to diagnose - and hopefully solve - this problem?
Thanks a lot!
mario

Are you sure you can access that private function? I'm no TS-guy but reads odd to me.

The noop function is a red herring. Sometimes it helps, sometimes not, so I'm forced to revert to plain http till something changes in koa.
Unfortunately I'm not able to be more specific in this issue: a small example does not exhibit the hang and even the full application does not consistently hangs.
(the private function works because everything related to the server is encapsulated in a class).
Anyway, when the rest of the application will be in a better shape, I'll retest this issue.
Thanks!
mario

Alright. Well, I'll close this for now then.

Crossing fingers, with node 14.15.1 and koa 2.13.1 my application seems to work with native http2 without problems.

Crossing fingers, with node 14.15.1 and koa 2.13.1 my application seems to work with native http2 without problems.

had the same issue, resolved by upgrading koa @crystalfp thanks for the heads up


hamburger popup takes 1-3 seconds to load menu

from October 2021 pre-launch bug bash
Created by: Justin
Description
on an older iPad (safari), when I click the hamburger, it takes 1-3 seconds for the menu to pop up
Info
Bug source: unknown
Browser: safari
PR: none
Attachments

@neuromusic can you see if this is still an issue please?

I can't reproduce this on mobile and it's pretty old, so closing. If it is still an issue please re-open or make a new one.


TypeError: Input type must be str

Traceback (most recent call last):
File "G:\project\project.py", line 6, in
result = extract_data('invoice/MyFlipkart.pdf',templates=templates)
File "C:\Users\Hp\AppData\Local\Programs\Python\Python37-32\lib\site-packages\invoice2data\main.py", line 93, in extract_data
return t.extract(optimized_str)
File "C:\Users\Hp\AppData\Local\Programs\Python\Python37-32\lib\site-packages\invoice2data\extract\invoice_template.py", line 178, in extract
output[k] = self.parse_date(res_find[0])
File "C:\Users\Hp\AppData\Local\Programs\Python\Python37-32\lib\site-packages\invoice2data\extract\invoice_template.py", line 113, in parse_date
value, date_formats=self.options['date_formats'], languages=self.options['languages']
File "C:\Users\Hp\AppData\Local\Programs\Python\Python37-32\lib\site-packages\dateparser\conf.py", line 89, in wrapper
return f(*args, **kwargs)
File "C:\Users\Hp\AppData\Local\Programs\Python\Python37-32\lib\site-packages\dateparser_init_.py", line 54, in parse
data = parser.get_date_data(date_string, date_formats)
File "C:\Users\Hp\AppData\Local\Programs\Python\Python37-32\lib\site-packages\dateparser\date.py", line 412, in get_date_data
raise TypeError('Input type must be str')
TypeError: Input type must be str

MyFlipkart.txt

date: Invoice Date:\s+(\d{1,2})\-(\d{1,2})\-(\d{2,4})

This regex is invalid as it captures three numbers. It results in an array of three entries. It must be a single string.
Fix your regex to use one capture only (one opening bracket and one closing bracket).


Switch from the archived jung-kurt PDF library to phpdave11's fork.

I'm trying to remove some unmaintained, archived repos from my dependency
tree, and this one is getting pulled in via a few other hops.
Kubernetes depends on gonum/plot, which depends on this library (go-latex/latex).
I already updated the pdf dependency for gonum/plot in https://github.com/gonum/plot/pull/654.

Thanks.
Could you add another commit adding yourself to the A+C files?
Thanks a lot.

Could you add another commit adding yourself to the A+C files?

All set! I added the commit here in this PR, please let me know if you'd prefer another one.

thanks. (merged with a slight change in the commit message, w/ 94de131)


Coordinate code update

This PR will deal with the changes in the codes related to coordinate systems that have been developed previously. The folder some_basic_coordinate_systems will be modified under this PR.

This is related to issue #6


Example of multiplexing ssh-over-tls to different hosts

I've got an example of multiplexing SSH over HTTPS to different hosts.
If there's a way to maintain the same functionality, but simplify the config, I'd love to know how to do it.
Is there some sort of template system I could use to say "here's a ruleset" and then "apply that ruleset, but for these hostnames"?
I believe the caddyfile has something like that.
1. Install xcaddy

Download xcaddy
make it executable
put it in your PATH
(pathman can help if you're unfamiliar with that)

pushd /tmp/

curl -o ./xcaddy_0.3.2_linux_amd64.tar.gz \
    -L https://github.com/caddyserver/xcaddy/releases/download/v0.3.2/xcaddy_0.3.2_linux_amd64.tar.gz

tar xvf ./xcaddy_0.3.2_linux_amd64.tar.gz

chmod a+x caddy

mkdir -p ~/bin/
mv ./caddy ~/bin/caddy

popd

2. Build with layer4
Needs layer4, l4tls, l4ssh, l4proxy.
We'll be using duckdns for real TLS certs since many clients don't issue SNI at all in -k, --insecure mode, and therefore cannot test against self-issued certs without putting the cert in the client's chain.
#!/bin/sh

export XCADDY_SETCAP=1
#export XCADDY_SUDO=0
export XCADDY_SKIP_CLEANUP=1

xcaddy build \
    --with github.com/mholt/caddy-l4/layer4 \
    --with github.com/mholt/caddy-l4/modules/l4tls \
    --with github.com/mholt/caddy-l4/modules/l4subroute \
    --with github.com/mholt/caddy-l4/modules/l4http \
    --with github.com/mholt/caddy-l4/modules/l4ssh \
    --with github.com/mholt/caddy-l4/modules/l4proxy \
    --with github.com/caddy-dns/duckdns

2b. How to run
Assuming an .env with the DUCKDNS_API_TOKEN:
caddy run --envfile ./.env --config ./config.json

3. Multiplexing HTTPS and SSH
Configure SSH Client
You can use openssl s_client, socat, sclient, or one of many other tls-terminating tools. Demonstrating with sclient because it is purpose-built for this use case and works on Windows.
~/.ssh/config:
Host example-a
    Hostname example-a.duckdns.org
    ProxyCommand sclient %h

Host example-b
    Hostname example-b.duckdns.org
    ProxyCommand sclient %h

Configure Caddy

example-a.duckdns.org

TLS is terminated
ssh goes to localhost on port 22
http goes to localhost http on port 3000

host header match to prevent SNI/Host mismatch attack

the same is true for exampl-b.duckdns.org

This feels a little too verbose. Is there a some sort of ruleset that could be created and then applied per-host?
{
  "logging": {
    "logs": {
      "default": {
        "level": "DEBUG"
      }
    }
  },
  "apps": {
    "layer4": {
      "servers": {
        "my-sshttp": {
          "listen": ["<IP_ADDRESS>:443"],
          "routes": [
            {
              "match": [
                {
                  "tls": {}
                }
              ],
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "match": [
                        {
                          "tls": {
                            "sni": ["example-a.duckdns.org"]
                          }
                        }
                      ],
                      "handle": [
                        {
                          "handler": "tls",
                          "connection_policies": [
                            {
                              "alpn": ["http/1.1"]
                            }
                          ]
                        },
                        {
                          "handler": "subroute",
                          "routes": [
                            {
                              "match": [{ "ssh": {} }],
                              "handle": [
                                {
                                  "handler": "proxy",
                                  "upstreams": [{ "dial": ["localhost:22"] }]
                                }
                              ]
                            },
                            {
                              "match": [
                                {
                                  "http": [
                                    { "host": ["example-a.duckdns.org"] }
                                  ]
                                }
                              ],
                              "handle": [
                                {
                                  "handler": "proxy",
                                  "upstreams": [{ "dial": ["localhost:3000"] }]
                                }
                              ]
                            }
                          ]
                        }
                      ]
                    },
                    {
                      "match": [
                        {
                          "tls": {
                            "sni": ["example-b.duckdns.org"]
                          }
                        }
                      ],
                      "handle": [
                        {
                          "handler": "tls",
                          "connection_policies": [
                            {
                              "alpn": ["http/1.1"]
                            }
                          ]
                        },
                        {
                          "handler": "subroute",
                          "routes": [
                            {
                              "match": [{ "ssh": {} }],
                              "handle": [
                                {
                                  "handler": "proxy",
                                  "upstreams": [{ "dial": ["<IP_ADDRESS>:22"] }]
                                }
                              ]
                            },
                            {
                              "match": [
                                {
                                  "http": [
                                    { "host": ["example-b.duckdns.org"] }
                                  ]
                                }
                              ],
                              "handle": [
                                {
                                  "handler": "proxy",
                                  "upstreams": [{ "dial": ["<IP_ADDRESS>:3000"] }]
                                }
                              ]
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      }
    },
    "tls": {
      "certificates": {
        "automate": [
          "localhost",
          "<IP_ADDRESS>",
          "example-a.duckdns.org",
          "example-b.duckdns.org"
        ]
      },
      "automation": {
        "policies": [
          {
            "subjects": ["localhost", "<IP_ADDRESS>"],
            "issuers": [
              {
                "module": "internal"
              }
            ]
          },
          {
            "subjects": ["example-a.duckdns.org", "example-b.duckdns.org"],
            "issuers": [
              {
                "challenges": {
                  "dns": {
                    "provider": {
                      "api_token": "{env.DUCKDNS_API_TOKEN}",
                      "name": "duckdns"
                    }
                  }
                },
                "module": "acme"
              },
              {
                "challenges": {
                  "dns": {
                    "provider": {
                      "api_token": "{env.DUCKDNS_API_TOKEN}",
                      "name": "duckdns"
                    }
                  }
                },
                "module": "zerossl"
              }
            ]
          }
        ]
      }
    }
  }
}

TODO
How to pass HTTP traffic through to Caddy's normal http handler? Perhaps related to https://github.com/mholt/caddy-l4/issues/70 and https://github.com/mholt/caddy-l4/pull/78?
How to simplify?

@mholt Any idea how to get that to listen on port 80 and respond to http strategy ACME challenges?

Use the http app to do that.


Persist dark mode on supported API levels

Codecov Report

Merging #866 (27f1acf) into main (6592ba0) will decrease coverage by 0.05%.
The diff coverage is 25.00%.

@@            Coverage Diff             @@
##             main     #866      +/-   ##
==========================================
- Coverage   62.88%   62.84%   -0.05%
==========================================
  Files         148      148
  Lines        7243     7251       +8
  Branches     1406     1410       +4
==========================================
+ Hits         4555     4557       +2
- Misses       1787     1793       +6
  Partials      901      901

Impacted Files
Coverage Δ

...exvanyo/composelife/appcompatsync/AppCompatSync.kt
18.18% <25.00%> (+3.89%)
:arrow_up:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


One user login to jupyterhub but others get internal server error.

Hi,
I'm new in this area and trying to install jupyterhub using sudospawner on my local machine to see how it works. Everything works fine for one user (the one I installed everything by that) but for other users I'm getting 500 internal server error.  It seems there is a permission issue, but I tried every solution I've found to fix it but nothing has worked.
Ubuntu 16.4
Jupyterhub 0.9.0.b1
Any idea what should I do?
I would be happy to give more information if needed.
Thanks,

Please include the output from jupyterhub when the failure occurs.
When it works for one user and not others, most often the issue is one of permissions and/or PATH. All users must be able to launch the jupyterhub-singleuser entrypoint and import all of the modules in that Python environment. This generally means read and execute permissions on the environment.
Typically, JupyterHub should be installed in a shared, system-wide installation, e.g. default /usr/local, or a dedicated env somewhere that should not be in anyone's home directory.
One way to test this is:
sudo -u some-other-user /path/to/jupyterhub-singleuser -h

It should show you the help output and not any errors about permissions.

Thank you for the response,
Here is the log when user test tries to login:
[I 2018-05-25 09:47:31.299 JupyterHub app:988] Loading cookie_secret from /home/fatemeh/jupyterhub/jupyterhub_cookie_secret
[I 2018-05-25 09:47:31.310 JupyterHub proxy:429] Generating new CONFIGPROXY_AUTH_TOKEN
[W 2018-05-25 09:47:31.313 JupyterHub configurable:168] Config option delete_invalid_users not recognized by PAMAuthenticator.
[I 2018-05-25 09:47:31.331 JupyterHub app:1162] Not using whitelist. Any authenticated user will be allowed.
[W 2018-05-25 09:47:31.341 JupyterHub app:1463] Deleting OAuth client jupyterhub-user-fatemeh
[W 2018-05-25 09:47:31.344 JupyterHub app:1463] Deleting OAuth client jupyterhub-user-test
[I 2018-05-25 09:47:31.359 JupyterHub app:1760] Hub API listening on http://<IP_ADDRESS>:8081/hub/
[W 2018-05-25 09:47:31.360 JupyterHub proxy:481] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
[I 2018-05-25 09:47:31.360 JupyterHub proxy:483] Starting proxy @ http://<IP_ADDRESS>:8000
09:47:31.480 - info: [ConfigProxy] Proxying http://<IP_ADDRESS>:8000 to (no default)
09:47:31.482 - info: [ConfigProxy] Proxy API at http://<IP_ADDRESS>:8001/api/routes
[I 2018-05-25 09:47:31.813 JupyterHub proxy:299] Checking routes
[I 2018-05-25 09:47:31.814 JupyterHub proxy:368] Adding default route for Hub: / => http://<IP_ADDRESS>:8081
09:47:31.814 - info: [ConfigProxy] 200 GET /api/routes
09:47:31.860 - info: [ConfigProxy] Adding route / -> http://<IP_ADDRESS>:8081
09:47:31.862 - info: [ConfigProxy] 201 POST /api/routes/
[I 2018-05-25 09:47:31.863 JupyterHub app:1817] JupyterHub is now running at http://<IP_ADDRESS>:8000
[I 2018-05-25 09:47:41.674 JupyterHub base:470] User logged in: test
[I 2018-05-25 09:47:41.675 JupyterHub log:158] 302 POST /hub/login?next= -> /user/test/<EMAIL_ADDRESS>17.88ms
[I 2018-05-25 09:47:41.681 JupyterHub log:158] 302 GET /user/test/ -> /hub/user/test/ (@<IP_ADDRESS>) 0.50ms
sh: 1: cannot create /run/motd.dynamic.new: Permission denied
The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.
The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.
[W 2018-05-25 09:47:41.795 JupyterHub auth:642] Failed to open PAM session for test: [PAM Error 14] Cannot make/remove an entry for the specified session
[W 2018-05-25 09:47:41.795 JupyterHub auth:643] Disabling PAM sessions from now on.
here ['sudo', '-u', 'test']
Traceback (most recent call last):
File "/usr/local/bin/sudospawner", line 4, in
import('pkg_resources').require('sudospawner==0.6.0.dev0')
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 2927, in
@_call_aside
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 2913, in _call_aside
f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 2940, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 635, in _build_master
ws.require(requires)
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 943, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 829, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'notebook' distribution was not found and is required by sudospawner
[E 2018-05-25 09:47:42.003 JupyterHub spawner:84] Failed to get JSON result from mediator: ''
[E 2018-05-25 09:47:42.003 JupyterHub user:472] Unhandled error starting test's server: substring not found
[E 2018-05-25 09:47:42.029 JupyterHub spawner:709] post_stop_hook failed with exception: <sudospawner.spawner.SudoSpawner object at 0x7f7617e58fd0>
Traceback (most recent call last):
File "/home/fatemeh/jupyterhub/jupyterhub/spawner.py", line 707, in run_post_stop_hook
return self.post_stop_hook(self)
TypeError: 'NoneType' object is not callable
[E 2018-05-25 09:47:42.031 JupyterHub web:1621] Uncaught exception GET /hub/user/test/ (<IP_ADDRESS>)
HTTPServerRequest(protocol='http', host='<IP_ADDRESS>:8000', method='GET', uri='/hub/user/test/', version='HTTP/1.1', remote_ip='<IP_ADDRESS>')
Traceback (most recent call last):
File "/home/fatemeh/.local/lib/python3.5/site-packages/tornado/web.py", line 1543, in _execute
result = yield result
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
result = coro.throw(exc)
File "/home/fatemeh/jupyterhub/jupyterhub/handlers/base.py", line 953, in get
await self.spawn_single_user(user)
File "/home/fatemeh/jupyterhub/jupyterhub/handlers/base.py", line 642, in spawn_single_user
await gen.with_timeout(timedelta(seconds=self.slow_spawn_timeout), finish_spawn_future)
File "/usr/lib/python3.5/asyncio/futures.py", line 361, in iter
yield self  # This tells Task to wait for completion.
File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
result = coro.send(None)
File "/home/fatemeh/jupyterhub/jupyterhub/handlers/base.py", line 597, in finish_user_spawn
await spawn_future
File "/home/fatemeh/jupyterhub/jupyterhub/user.py", line 484, in spawn
raise e
File "/home/fatemeh/jupyterhub/jupyterhub/user.py", line 404, in spawn
url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
File "/usr/lib/python3.5/asyncio/futures.py", line 361, in iter
yield self  # This tells Task to wait for completion.
File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/home/fatemeh/jupyterhub/sudospawner/sudospawner/spawner.py", line 93, in start
reply = yield self.do(action='spawn', args=self.get_args(), env=self.get_env())
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/home/fatemeh/jupyterhub/sudospawner/sudospawner/spawner.py", line 81, in do
data_str = data_str[data_str.index('{'):data_str.rindex('}')+1]
ValueError: substring not found
[E 2018-05-25 09:47:42.080 JupyterHub log:150] {
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8",
"Accept-Encoding": "gzip, deflate",
"X-Forwarded-Port": "8000",
"X-Forwarded-For": "<IP_ADDRESS>",
"Upgrade-Insecure-Requests": "1",
"X-Forwarded-Proto": "http",
"Referer": "http://<IP_ADDRESS>:8000/hub/login",
"Connection": "close",
"Accept-Language": "en-GB,en;q=0.5",
"Cookie": "jupyterhub-hub-login="2|1:0|10:1527256061|20:jupyterhub-hub-login|44:YTZmZDc2OWMyZmNmNDgyNzk0NGM3MmJiM2MxNWIxZTU=|27089f4ae37e9903d090cd37b285585632e5f5e217324ddd494c64cd0e3bde45"; _xsrf=2|0637f70a|be46b6c98aa82d32a2947955876744db|1525883715; jupyterhub-session-id=639ec2f9562740be8ca66318f46b5a9a",
"User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0",
"X-Forwarded-Host": "<IP_ADDRESS>:8000",
"Host": "<IP_ADDRESS>:8000"
}
[E 2018-05-25 09:47:42.080 JupyterHub log:158] 500 GET /hub/user/test/<EMAIL_ADDRESS>379.21ms
Also, when I run the command "sudo -u some-other-user /path/to/jupyterhub-singleuser -h" for this user I'm getting this error:
Traceback (most recent call last):
File "/usr/local/bin/jupyterhub-singleuser", line 4, in
import('pkg_resources').require('jupyterhub==0.9.0b1')
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 2927, in
@_call_aside
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 2913, in _call_aside
f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 2940, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 635, in _build_master
ws.require(requires)
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 943, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 829, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'prometheus_client>=0.0.21' distribution was not found and is required by jupyterhub
All jupyterhub, sudospawner and jupyterhub-single user are in /usr/local/bin/ path. and here is the permissions:
-rwxr-xr-x 1 root root 222 May 11 22:49 /usr/local/bin/jupyter
-rwxr-xr-x 1 root root 236 May 11 22:49 /usr/local/bin/jupyter-bundlerextension
-rwxr-xr-x 1 root root 289 May 17 18:15 /usr/local/bin/jupyterhub
-rwxr-xr-x 1 root root 311 May 17 18:15 /usr/local/bin/jupyterhub-singleuser
-rwxr-xr-x 1 root root 226 May 11 22:49 /usr/local/bin/jupyter-kernel
-rwxr-xr-x 1 root root 264 May 11 22:49 /usr/local/bin/jupyter-kernelspec
-rwxr-xr-x 1 root root 222 May 11 22:49 /usr/local/bin/jupyter-migrate
-rwxr-xr-x 1 root root 224 May 11 22:49 /usr/local/bin/jupyter-nbconvert
-rwxr-xr-x 1 root root 223 May 11 22:49 /usr/local/bin/jupyter-nbextension
-rwxr-xr-x 1 root root 222 May 11 22:49 /usr/local/bin/jupyter-notebook
-rwxr-xr-x 1 root root 243 May 11 22:49 /usr/local/bin/jupyter-run
-rwxr-xr-x 1 root root 227 May 11 22:49 /usr/local/bin/jupyter-serverextension
-rwxr-xr-x 1 root root 227 May 11 22:49 /usr/local/bin/jupyter-troubleshoot
-rwxr-xr-x 1 root root 255 May 11 22:49 /usr/local/bin/jupyter-trust
-rwxr-xr-x 1 root root 315 May 17 18:15 /usr/local/bin/sudospawner

I also changed the group of jupyter files in /usr/local/bin to jupyterhub (which is the group of jupyterhub users), but still getting error 500.
-rwxr-xr-x 1 root jupyterhub 222 May 11 22:49 /usr/local/bin/jupyter
-rwxr-xr-x 1 root jupyterhub 236 May 11 22:49 /usr/local/bin/jupyter-bundlerextension
-rwxr-xr-x 1 root jupyterhub 289 May 17 18:15 /usr/local/bin/jupyterhub
-rwxr-xr-x 1 root jupyterhub 311 May 17 18:15 /usr/local/bin/jupyterhub-singleuser
-rwxr-xr-x 1 root jupyterhub 226 May 11 22:49 /usr/local/bin/jupyter-kernel
-rwxr-xr-x 1 root jupyterhub 264 May 11 22:49 /usr/local/bin/jupyter-kernelspec
-rwxr-xr-x 1 root jupyterhub 222 May 11 22:49 /usr/local/bin/jupyter-migrate
-rwxr-xr-x 1 root jupyterhub 224 May 11 22:49 /usr/local/bin/jupyter-nbconvert
-rwxr-xr-x 1 root jupyterhub 223 May 11 22:49 /usr/local/bin/jupyter-nbextension
-rwxr-xr-x 1 root jupyterhub 222 May 11 22:49 /usr/local/bin/jupyter-notebook
-rwxr-xr-x 1 root jupyterhub 243 May 11 22:49 /usr/local/bin/jupyter-run
-rwxr-xr-x 1 root jupyterhub 227 May 11 22:49 /usr/local/bin/jupyter-serverextension
-rwxr-xr-x 1 root jupyterhub 227 May 11 22:49 /usr/local/bin/jupyter-troubleshoot
-rwxr-xr-x 1 root jupyterhub 255 May 11 22:49 /usr/local/bin/jupyter-trust
-rwxr-xr-x 1 root jupyterhub 315 May 17 18:15 /usr/local/bin/sudospawner

Try checking the permissions in the dist-packages and/or site-packages directories. When you import the notebook package as the user that works, check that the permissions on that package make it readable by all users.


Netty classloading duplicate calls to NetworkInterface.getNetworkInterfaces()

Expected behavior
Can we avoid this duplicate call by caching NetUtil loaded network interfaces
and adapt MacAddressUtil to call NetUtil.getNetworkInterfaces() ?
Actual behavior
MacAddressUtil,bestAvailableMac() method is calling once NetworkInterface.getNetworkInterfaces()
But just before its triggering also NetUtil class loading that's invoking static method where NetworkInterface.getNetworkInterfaces() is called.

It's causing slowness for us during Grpc bootstrap process.
Steps to reproduce
Call MacAddressUtil,bestAvailableMac()
Minimal yet complete reproducer code (or URL to code)
Netty version
Latest
JVM version (e.g. java -version)
1.8
OS version (e.g. uname -a)
Windows 10

Should be "fixed" by https://github.com/netty/netty/pull/13060

There is no planed date yet


Fixed dependencies

Some dependencies had dev-master versions, but in meantime, new major versions were released

If these changes haven't been tested I don't think we can merge this


Ensure clipboard.js instance is destroyed when component is torndown

This simply cleans up the clipboard instances so we don't end up with
a bunch of them in memory after our components are removed from the DOM.

Looks like tests are failing because jquery is not pinned. See https://github.com/ember-cli/ember-cli/pull/5327.

I guess there's some other Ember CLI dependency issues too... :frowning:


Update core-js to allow removing a structuredClone work-around

Because of a bug in previous core-js versions, which caused an Error to be thrown if its structuredClone polyfill was called with an explicit null/undefined transfer-parameter, the LoopbackPort-class contained a work-around.
In the latest core-js version this has been fixed, and we can thus simplify our code ever so slightly; please see https://github.com/zloirock/core-js/releases/tag/v3.22.0

/botio unittest

Much better indeed!


FTT-Freight turnover rate inconsistency

In the master file, the turnover rate for FTT-Fr is given as 10 years for all technologies. However, in the region CSV files, this number is 1.75. This is quite a big difference and impacts the dynamics of the model quite a lot.
Given the turnover rate in FTT-Tr is 4 years and I would imagine freight to be more than this, I'm going to use the 10 year value for now.
Which of these is the correct value?

Thanks for the question. Are you sure this is a turnover rate number? If the turnover rate is 1.75, this must be an error. In general, freight turnover rate is closed to cars because the mechanical lifetime of a truck is around 10-20 years.

Thanks Aileen, good to know that is an error. Do you think 10 years is too high then? Should we change this number to be the same as for cars?


[Feature] Quantization global optimization

ptq quantized and deployed experiment results on openvino and trt backends
mainly changes:

add mmcls and mmdet models ptq configs on tensorrt
change ptq calibration dataset from train to val
unify calibration batch size and calibration steps
replace fakequant type and freeze params before deployment
optimaze handle pipeline for w_fake_quant before deployment
optimaze tensorrt backendconfig
optimaze interfaces of quantizers

release下对过精度的onnx文件？


Rewrite data download and tokenization pipeline

Now supports:

Lmd based archive handling (this includes efficient zstd and tar handling)
No more unpacking compressed files
Multi-file support
Vastly simplified dataset class

I'm getting this error when processing 'PileSubset' - I'm guessing it's encountering an empty document or something?
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "tools/preprocess_data.py", line 98, in encode
    doc_ids[-1].append(Encoder.tokenizer.eod)
IndexError: list index out of range
"""

Added a filter for empty documents

Ok great, I'm going to add a few more pile datasets before we merge. Can lmd to tar.gz?

It should be able to, and if there's any format it can't do that we need I'll update lmd to support it

Also, do we want to have the urls point to eaidata, or the eye?


Improve Analyzer Visibility

This pull request introduces changes to enhance the visibility of the analyzer.
Purpose of Change

Improve the visibility of the analyzer.

Changes

The user can select the visibility, visible or hidden, of the waveforms and the spectrograms individually.

The user can easily compare the waveforms or the spectrograms.
This option can be set through the setting file as well as GUI.

The user can set the height scale of the waveforms and the spectrograms individually.

It allows the user to observe multi-channel data.
The scale can be set through the setting file.

Those axes are drawn in interval of 1, 2, 5 or the powers of 10 each.
for the time axis:

It allows the user to easily confirm the current position.
The user can easily seek the position they want to play.

for the waveform vertical axis:

The axis always has the value of zero even if the waveform is not symmetric.

Current axes are also available through the setting file.

Add channel labels for each of the waveforms and the spectrograms.

This option can be set through the setting file.

Settings and Defaults
The following settings are added in this PR.

waveformVisible, spectrogramVisible: boolean; default = true

Set the visibility of the waveforms and the spectrograms.

waveformVerticalScale, spectrogramVerticalScale: number; default = 1.0

Set the height scale of the waveforms and the spectrograms.

waveformShowChannelLabel, spectrogramShowChannelLabel: boolean; default = false

Set the visibility of the channel labels for the waveforms and the spectrograms.

roundTimeAxis: boolean; default = proposed to be true

By making this true, the horizontal axes interval is set to 1, 2, 5 or the powers of 10 each.
I have made the original axis drawing algorithm available, but I believe enabling this feature will help the majority of users.

roundWaveformAxis: boolean; default = proposed to be true

By making this true, the vertical axes interval of the waveform is set to 1, 2, 5 or the powers of 10 each.
I have made the original axis drawing algorithm available, but I believe enabling this feature will help the majority of users.

Tests

I've confirmed the change visually.
Added a test to assess the newly defined function, roundToNearestNiceNumber (analyzeService.test.ts).
Some tests have been modified to handle newly added settings.
The changes conform to all existing tests, including added or modified test.

Note that this is the last PR in a series of my recent modifications.

I'm sorry for my late response.
Thank you very much for the detailed review.
I have updated the code based on your suggestions.
Could you please check it?
Regarding WAVEFORM_CANVAS_VERTICAL_SCALE_MIN, the round axis algorithm initially didn't work well and failed to draw any axes if the minimum canvas height was too small.
Now, the algorithm has been fixed, and it seems that at least one axis is drawn even if the vertical scale is 0.2.
Therefore, I agree to change WAVEFORM_CANVAS_VERTICAL_SCALE_MIN to 0.2.


call generateUniqueIdentifier once per added file

addFiles calls generateUniqueIdentifier, but then FlowFile also calls generateUniqueIdentifier. Not a big deal, but an inefficiency nonetheless.

I found this because I was using generateUniqueIdentifier to request a unique identifier from the server, and flow was performing two successive GETs.


Is Cron event supported?

This doc describes "Container Apps can scale based of a wide variety of event types. Any event supported by KEDA, is supported in Container Apps". So, Cron event should be supported. But it doesn't work in my environment.
$ az extension show -n containerapp
ExtensionType    Name          Path                                              Version
---------------  ------------  ------------------------------------------------  ---------
whl              containerapp  /home/me/.azure/cliextensions/containerapp  0.2.0

I ran az containerapp create with --scale-rules option. and the following is the rule.
[
  {
    "name": "cron",
    "custom": {
      "type": "cron",
      "metadata": {
        "timezone": "Asia/Tokyo",
        "start": "15 * * * *",
        "end": "20 * * * *",
        "desiredReplicas": "1"
      }
    }
  }
]

Then, I got the error.
--scale-rules: type field required in rules config

So, I edited the rule as follows. And ran it again.
[
  {
    "name": "cron",
    "type": "cron",
    "metadata": {
      "timezone": "Asia/Tokyo",
      "start": "15 * * * *",
      "end": "20 * * * *",
      "desiredReplicas": "1"
    }
  }
]

This time my container app was created without error. However, the rules did not seem to be reflected. The following is resource status of my container app. There is no rule I specified.
...
            "scale": {
                "minReplicas": 1,
                "maxReplicas": 1
            }
        }
...

Is Cron event supported? I would appreciate it if you could check it.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 58fa8bbe-e0eb-4944-bc60-617aec0c0dde
Version Independent ID: eae53948-3193-a451-5963-3e6cd8ee2972
Content: Scaling in Azure Container Apps
Content Source: articles/container-apps/scale-app.md
Service: container-apps
GitHub Login: @craigshoemaker
Microsoft Alias: cshoe

Hello @ToruMakabe - Thank you for your feedback! We will review and update as appropriate.

@mike-urnun-msft I have tried a deployment of Cron scaler again with ARM template instead of Azure CLI, then it was successful. This may be a problem of Azure CLI extension.
It's not a documentation problem, so I will close it. Thank you for your support.


[🐞] v2: repl somewhat broken

Which component is affected?
Qwik Playground
Describe the bug
in v2 the repl no longer shows the html etc output tabs, and initial console logging seems broken too
Reproduction
https://qwikdev-build-v2.qwik-8nx.pages.dev/playground
Steps to reproduce
click on the html tab in the output panel
System Info
browser

Additional Information
No response

Fixed by #7170 and #7171


[BEAM-8956] Begin unifying contributor instructions into a single location

We have overlapping and sometimes contradictory docs on how to setup and build BEAM in four different places I've found:
README.md
CONTRIBUTING.md
https://cwiki.apache.org/confluence/display/BEAM/Contributor+FAQ
https://beam.apache.org/contribute/
We should probably pick one as the source of truth and rewrite the
other three to simply point to it. I propose putting all checkout,
build, test, commit, and push instructions in CONTRIBUTING.md in the
repo.
@iemejia
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[X] Choose reviewer(s) and mention them in a comment (R: @username).
[X] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[X] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

R: @iemejia

PTAL. Now unified into website.
FYI, I have not yet figured out how to build/run the website locally. Something about my gLinux docker setup is off:

Task :website:buildDockerImage FAILED
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.40/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&session=zsx1afhzf5weazb0n9n456xgb&shmsize=0&t=beam-website&target=&ulimits=null&version=1: dial unix /var/run/docker.sock: connect: permission denied

Seems like you need to configure Docker in "sudoless"  mode, i.e., give your user access to that socket file.

@iemejia @kennknowles could you please review this PR?

Done, merged manually. Thanks for the reminder @aaltay  and sorry @elharo for taking so long on merging this. I suppose next step is to move to the wiki, is it?
In any case thanks for taking care of this.

Ah, sorry. I somehow failed to send my comment, which was that we should keep CONTRIBUTING.md just to have a pointer to the primary instructions. This file is treated specially by GitHub. It is not terribly important. I don't know the real impact.


Bump version of used github action run-clang-format to v1.1.0

Describe your changes

Bump version of used github action run-clang-format to v1.1.0

Issue ticket number and link
https://github.com/EVerest/everest-ci/issues/11
Checklist before requesting a review

[x] I have performed a self-review of my code
[ ] I have made corresponding changes to the documentation
[x] I read the contribution documentation and made sure that my changes meet its requirements

Can't merge seems to need approval from all reviewers, shouldn't be only one approval necessary @Pietfried ?

shouldn't be only one approval necessary

Probably one of each set of code owners for each file affected is required. The reformatting touched quite a lot of modules, so all their approvals might be necessary.
https://github.com/EVerest/everest-core/blob/f1ad5ccf33de21e7cb507f6e421bfd3930534630/.github/CODEOWNERS#L18-L33
I guess YetiDriver code owners are not covered yet.

@barsnick that makes sense, thank you for explaining.


prepDE.py Error: Text file with sample ID and path invalid ()

Hi,
I was trying to extract count matrix by prep.DE, but I was getting the error.
Error: Text file with sample ID and path invalid ()

This is the command I used,
$ python prepDE.py -i sample_lst_2M.txt -g ./gene_count_matrix_2M.csv -t ./transcript_count_matrix_2M.csv

This is how the sample_lst_2M.txt looks like.
Ctrl1	/home/IrisC/Hisat2/Stringtie/S0311-4h-c_L1/Ctrl1.gtf
Ctrl2	/home/IrisC/Hisat2/Stringtie/S0316-4h-c_L1/Ctrl2.gtf
2M1B1	/home/IrisC/Hisat2/Stringtie/S0311-4h-2_L1/2M1B1.gtf
2M1B2	/home/IrisC/Hisat2/Stringtie/S0316-4h-2_L1/2M1B2.gtf

The directory structure:
$ tree -L 3 /home/IrisC/Hisat2/Stringtie/

Stringtie/
├── DE
│   └── sample_lst_2M.txt
├── S0311-4h-2_L1
│   ├── 2M1B1.gtf
│   ├── e2t.ctab
│   ├── e_data.ctab
│   ├── i2t.ctab
│   ├── i_data.ctab
│   ├── S0311-4h-2_L1_gene_abund.tab
│   └── t_data.ctab
├── S0311-4h-c_L1
│   ├── Ctrl1.gtf
│   ├── e2t.ctab
│   ├── e_data.ctab
│   ├── i2t.ctab
│   ├── i_data.ctab
│   ├── S0311-4h-c_L1_gene_abund.tab
│   └── t_data.ctab
├── S0316-4h-2_L1
│   ├── 2M1B2.gtf
│   ├── e2t.ctab
│   ├── e_data.ctab
│   ├── i2t.ctab
│   ├── i_data.ctab
│   ├── S0316-4h-2_L1_gene_abund.tab
│   └── t_data.ctab
├── S0316-4h-c_L1
│   ├── Ctrl2.gtf
│   ├── e2t.ctab
│   ├── e_data.ctab
│   ├── i2t.ctab
│   ├── i_data.ctab
│   ├── S0316-4h-c_L1_gene_abund.tab
│   └── t_data.ctab
└── Read.md

I have tried executing prepDE.py by the following command but it didn't fix the error.
$ python2.7 prepDE.py -i sample_lst_2M.txt -g ./gene_count_matrix_2M.csv -t ./transcript_count_matrix_2M.csv

Does anyone know how to fix this?
Any suggestion would be appreciated.
Thanks,
Iris

Hi,
I faced a similar issue. I found that there was an empty line at the bottom of the .txt file. Removing that empty line fixed my problem.
You can check your txt file.

Hi,
I faced a similar issue. I found that there was an empty line at the bottom of the .txt file. Removing that empty line fixed my problem.
You can check your txt file.

Yes, I had the same problem. Thanks very much!!!

I also have same problem.


Update appsettings.json to increase Auth duration

This will also need to be changed in the User secrets file as it will overide


Masks And Detection on Inference

Hi Everyone & @dusty-nv ,
I have successfully tested segmentation using usb cam and masks overlay and detection seems working fine on output frames but I am unable to get the masks and detection as return which I want to use it for post processing.
I have checked the API reference link for segnet as well and it says all the functions such as
net.Overlay(img_overlay, width, height, opt.filter_mode)
net.Mask(img_mask, width/2, height/2, opt.filter_mode)
etc
https://rawgit.com/dusty-nv/jetson-inference/pytorch/docs/html/python/jetson.inference.html#segNet
Almost all function's having None as returned type. So, please any help to get the detection boxes and masks points as return type ?
I have seen there's option in object detection but there is so far couldn't find any option in segmentation.
I will be very thank you.
FileName: segnet-camera.py.
net = jetson.inference.segNet(opt.network, sys.argv)
net.SetOverlayAlpha(opt.alpha)
img_overlay = jetson.utils.cudaAllocMapped(opt.width * opt.height * 4 * ctypes.sizeof(ctypes.c_float))
img_mask = jetson.utils.cudaAllocMapped(opt.width/2 * opt.height/2 * 4 * ctypes.sizeof(ctypes.c_float))
camera = jetson.utils.gstCamera(opt.width, opt.height, opt.camera)
display = jetson.utils.glDisplay()
while display.IsOpen():
img, width, height = camera.CaptureRGBA()
net.Process(img, width, height, opt.ignore_class)

net.Overlay(img_overlay, width, height, opt.filter_mode)
net.Mask(img_mask, width/2, height/2, opt.filter_mode)

display.BeginRender()
display.Render(img_overlay, width, height)
display.Render(img_mask, width/2, height/2, width)
display.EndRender()

display.SetTitle("{:s} | Network {:.0f} FPS".format(opt.network, net.GetNetworkFPS()))

It is possible to access masks by using the segmentation buffers, defined here.
You can then directly by using mask = buffers.mask after this line.
To convert the cudaImage to numpy array, you can use mask_np = jetson.utils.cudaToNumpy(mask)


Deprecate User Defined Variables, and hide it from the menus

Expected behavior
There should be a way to initialize variables at the beginning of the test.
For instance:
a) Initialize variables based on properties
b) Set thread-group specific variables
In JMeter, variable should have thread scope, and it is unexpected for the variables from different threads to interfere.
The name User Defined Variable suggests that the config element configures variables, however, it configures test plan variables which is completely misleading
Actual behavior
Currently, User Defined Variables are replaced globally in the test plan which results in surprising behaviour. For instance, if several User Defined Variables located under different thread groups share a variable name, only one of the values win, and the same value is reused for all the threads.
Steps to reproduce the problem

udv_sharing.zip
JMeter Version
5.5
Java Version
17
OS Version
No response

The behaviour of the User Defined Variables is documented:
https://jmeter.apache.org/usermanual/component_reference.html#User_Defined_Variables
UDV are initialized once at test start.
To make this behaviour more clear and to prevent errors, UDVs should be only allowed on "root" level. More than one UDV block could make sense in case some variables rely on others being set before.

Frankly speaking, that documentation and the behavior were obscure to me until I analyzed the sources.
At the same time, the reasons for that behavior are not clear either.

I would suggest not to deprecate the UDV, but to separate these.
Global UDV and thread scoped UDV.
That way, the scopes of both would be clear whilst remaining backward compatible.


GO:0009986 Cell surface: improve definition

Stemming from the discussion in https://github.com/geneontology/go-annotation/issues/1621
@ValWood kindly offered to propose an improved definition for 'cell surface'. A suggestion is to rename to 'external of the cell'.

Probably involves clarifying that cell surface only contains the external leaflet, so it should NOT apply to membrane spanning proteins.
Cell wall should be a child.

Hi All
I have gone back to looking at the ontology for this domain because I think we need something for curators to use to actually create annotations in a consistent way.
I just think it is time we created a figure (like the one Val drew in #14948) which shows how each different type of entity associated with the membrane and cell wall should be annotated and then to have this available on the http://geneontology.org/page/go-annotation-conventions site. This page exists: http://geneontology.org/page/membrane-proteins but it does not appear to be listed on the http://geneontology.org/page/go-annotation-conventions, so I am not sure how people can get to this page. I found it when looking for a suitable figure in Rebecca's GOC presentation from 2016.
While I like this figure and I think it is very helpful I don't think it actually states that if a protein is spanning the membrane then it should only be annotated to integral component of membrane ; GO:0016021, and not annotated to the following terms: cell surface, external side of plasma membrane, cytoplasmic side of plasma membrane. It would be useful to clarify whether a protein which is an integral component of membrane should also be annotated to these other 3 (or perhaps just the external and cytoplasmic side of plasma membrane) terms. And if this is how curators should be annotating plasma membrane terms shouldn't all terms such as GO:0008076   voltage-gated potassium channel complex have external and cytoplasmic side of plasma membrane terms as direct parent terms.
This would also encourage curators to consistently capture this information.
This figure also does not address the issue of the cell surface and cell wall.
Furthermore, what this figure does illustrate is the problem that arises if people are not allowed to take into account general biological knowledge. If a protein is known to have a membrane domain and is identified by FACs using an antibody that does not penetrate the cell then curators will select the GO term 'cell surface' or 'external side of plasma membrane' because that is what the experiment shows. If curators are encouraged to consider biological knowledge (for example using InterPro data or protein structure) which predicts the protein is integral to a membrane then more accurate grouping of proteins/complexes can be achieved.
Note InterPro often does not predict the specific membrane type and therefore the expt evidence with biological knowledge would lead to the creation of consistent annotations of all plasma membrane proteins to 'integral to plasma membrane' instead of the variety of annotations that currently exist for membrane spanning proteins.
Looking at the plasma membrane terms available I think there are too many and many are hardly  being used because they are almost impossible to experimentally demonstrate.
On slide 10 I point out that: The comment in the term GO:0044214 spanning component of plasma membrane: ‘Proteins that span the membrane but have the bulk on one side of the membrane may be additionally annotated with a term of the form integral to X side of the plasma membrane’ means that  a curator would be encouraged to annotate a receptor to integral to external side of the plasma membrane and thus annotate the receptor to cell surface.
However, how is a curator supposed to know if a protein has it’s bulk on one side or the other? Biological knowledge?
A more consistent approach would be that if the protein can be detected on one side or the other then we have to assume it is has some amount of it on that side. To have to make a judgement about whether this is sufficient to be considered its ‘bulk’ is just not feasible.
I have created a google document with slides which could be used to workout what we are trying to capture and what terms are needed. https://docs.google.com/presentation/d/1PjAp2pnEnPSiOPVUJAGovjUD0QnlUZv5tXprFmsZu9E/edit?usp=sharing
Please look at the last slide where I suggest considering removing some of these plasma membrane terms.
This doesn't try to resolve the cell surface issue, but I was hoping that if we look at the ontology it might help us work out what we already have and how the terms are being used.
Ruth

Would be useful, it's all a bit confusing...

Thanks for pulling this altogether, @RLovering  it makes it much easier to digest. I will look at this over next week and try to think of some demi-useful comments.
Based on my gut feeling, I think that the most important thing is that users see an unambiguous term - preferably one term that maps directly to one type of membrane association, but to achieve this utopia we need still to resolve the conflict about experimental observation vs inference. As we have so much more sequence-function knowledge now, perhaps we should unleash combinatorial evidence codes to allow us all to get past this constant sticking point (tagging @vanaukenk as an interested party).

I appreciate that we need to address the conflict about experimental observation vs inference, however, I do think it would help if we actually look at the ontology to make sure it describes what we think it describes. Certainly at least if we agree on the ontology we can look at how to improve annotation consistency.
Although, I am still of the opinion that 90% of annotations are based on previous knowledge (ie inferred) and that to have 10 terms which we apply in annotations without using previous knowledge is going to lead down the path of changing all IDA annotation to 'IDA+other info annotations'. In which case why not just go the other way and use the new ECO codes that are available when no previous knowledge is used, no inference is used but the annotation is actually based completely on experimental data. For example apply the very specific ECO code for Electron Microscopy, assuming that you trust the antibody as specific for detecting your protein, which of course is another inference......

I think that with your pictures we should be able to come up with a more limited palete of terms.
May be I am misinterpreting this, but my major concern is that we will end up with having to use multiple terms to describe one particular way of associating with the plasma membrane, particulary wrt proteins that span the membrane and possess intra and extracellular domains ending up with:
GO:0009898 cytoplasmic side of plasma membrane
GO:0009897 external side of plasma membrane
GO:0005887 integral component of plasma membrane
Which leads to inconsistencies and patchiness in annotation, which is confusing for users and curators.
There are a limited number of ways in which proteins associate with the plasma membrane and we should be able to map each one to a specific term. If the curator cannot tell which term/parent term, to use then they should fall back to “plasma membrane” rather than using a term that captures the limits of the assay.

I am unassigning myself from this ticket  because  it is beyond what I can do as an individual.
This might be a useful topic for an annotation call, or a GO meeting workshop activity?


WIP: Fly server

This adds another example server (fly-server.py) that uses fly.io's API to spin up a new Fly Machine for every new bot, instead of using a python subprocess in the web server itself. This will allow us to use Fly's new beefier GPU instances if we want to run local models in the cloud.
I'm putting this here now so I can hopefully remember to come back around to it before or after the hackathon. :)

This one is obsolete, closing. We have a working fly.io example here https://github.com/pipecat-ai/pipecat/tree/main/examples/deployment/flyio-example


more clearly separate the build time and run time configuration in the docs and scripts.

Some of this has been done by splitting the verification of build parameters from runtime parameters.

done.


[BEAM-6526] Remove unneeded methods in AvroIO ReadFiles and ParseFiles

This is a fix to an API issue I found in the previously introduced ReadFiles/ParseFiles PR.
I also included an extra commit to refactor AvroIO Read/Parse to use FileIO + ReadFiles/ParseFiles.
R: @chamikaramj

Run Java AvroIO Performance Test

Run Java PreCommit

CI is green now, PTAL @chamikaramj

Run Java PostCommit

LGTM. Thanks.


Records missing (there in FC1)

Searched for John McKendrick (born c 1801) could not be found, then tried his son Robert (born c 1830) again could not be found: However, both records can be found on FreeCEN1 (Piece: SCT1841/272 Place: Arbroath -Angus Enumeration District: 2) so I know the record exists, therefore there is a problem with the search on FreeCEN2.

@DeniseColbert what year?

I had no trouble locating on CEN2

I also found them OK on both search sites.
From: Kirk Dawson<EMAIL_ADDRESS>Sent: 09 December 2019 16:27
To: FreeUKGen/FreeCENMigration<EMAIL_ADDRESS>Cc: Subscribed<EMAIL_ADDRESS>Subject: Re: [FreeUKGen/FreeCENMigration] Records missing (there in FC1) (#768)
I had no trouble locating on CEN2
https://user-images.githubusercontent.com/2874694/70453461-0882d600-1a66-11ea-8710-ea33cb3e951e.JPG
https://user-images.githubusercontent.com/2874694/70453474-0de02080-1a66-11ea-9589-718b9a42b923.JPG
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub https://github.com/FreeUKGen/FreeCENMigration/issues/768?email_source=notifications&email_token=ADIL3VVGPGL5JZLMNO7UTDTQXZWVTA5CNFSM4JYIICF2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGJY42Y#issuecomment-563318379 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ADIL3VXFDFUBQNTR3KOZ333QXZWVTANCNFSM4JYIICFQ .

Brenda also told me offline that she found the record. Must have been an odd glitch! Will close.