7391 generate style properties on the fly based on wizard properties

ongoing

@xavijam: this code generates a JS error when the form tries to initialize the style animated form (this._animatedFormModel = new StyleAnimatedFormModel is in the stack trace). Error:
style-animated-properties-form-model.js:12 Uncaught TypeError: Cannot read property 'toString' of undefined

This is style_properties value:
{"autogenerated"=>true,
 "type"=>"simple",
 "properties"=>
  {"blending"=>"none",
   "fill"=>{"color"=>{"fixed"=>"#FF6600", "opacity"=>0.9}},
   "labels"=>
    {"attribute"=>nil,
     "font"=>"DejaVu Sans Book",
     "offset"=>-10,
     "overlap"=>true,
     "placement"=>"simple",
     "fill"=>{"size"=>{"fixed"=>10}, "color"=>{"fixed"=>"#000", "opacity"=>1}},
     "halo"=>{"size"=>{"fixed"=>1}, "color"=>{"fixed"=>"#FFF", "opacity"=>1}},
     "enabled"=>true}}}

As we talked, we have to add default values for animated (and labels).
{"autogenerated"=>true,
 "type"=>"simple",
 "properties"=>
  {"blending"=>"none",
   "fill"=>{"color"=>{"fixed"=>"#FF6600", "opacity"=>0.9}},
   "labels"=>…,
   "animated"=>
    {"attribute"=>nil,
     "duration"=>30,
     "overlap"=>false,
     "steps"=>"256",
     "resolution"=>"2",
     "trails"=>"2",
     "enabled"=>false}}}

@xavijam labels defaults (taken from the wiki, but with enabled: false):
      labels: {
          enabled: false,
          attribute: null, // text-name
          font: 'DejaVu Sans Book', // text-face-name
          fill: {
            'size': {
              fixed: 10 // text-size
            },
            'color': {
              fixed: '#000', // text-fill
              opacity: 1
            }
          },
          halo: {
            'size': {
              fixed: 1 // text-halo-radius
            },
            'color': {
              fixed: '#111', // text-halo-fill
              opacity: 1
            }
          },
          offset: -10, // text-dy
          overlap: true, // text-allow-overlap
          placement: 'point' // text-placement-type
      }

@juanignaciosl, yep ^^^ 👍

@javisantana Since there's no "cluster" mapping opening it with the new editor fails. This is the error:
POST http://juanignaciosl.localhost.lan:8181/api/v1/map?stat_tag=b2f1d664-20d1-11e6-80e3-080027880ca6&api_key=f5fe77ab2416fd66d4a6032d985b256fd411dd4e 400 (Bad Request)
map-base.js:80 Error creating the map instance on Windshaft: Postgis Plugin: ERROR:  column "points_count" does not exist
LINE 1: ..._webmercator",3.50861e-06), 7.01722e-06)) AS geom,"points_co...
                                                             ^

in executeQuery Full sql was: 'SELECT ST_AsBinary(ST_Simplify(ST_SnapToGrid("the_geom_webmercator",3.50861e-06), 7.01722e-06)) AS geom,"points_count","src" FROM (SELECT * FROM tac2016_stacked) as cdbq WHERE "the_geom_webmercator" && ST_MakeEnvelope(-20037508.3,20037508.25881302,-20037508.25881302,20037508.3,3857) LIMIT 65535'

Have you planned any strategy for cluster wizards?

No, cluster wizards should be:

marked as custom
move the query_wrapper to sql_wrap

@gfiorav CR this, please.
@xavijam I haven't found major issues (other than missing intensity wizard mapping). This is deployed to ded01, could you take a look to check that if I merge this it won't block you?

Not much to add:
Nice and thorough tests. 📦 it!

@gfiorav thanks!
@xavijam this is deployed in ded01 in case you want to take a look. I haven't found any major issue. I'll deploy it tomorrow morning.


actions are buffered when window looses focus

Godot version:
3.1 stable
OS/device including version:
tested on Win and Linux.
Issue description:
When an action is triggered and held within a _process function and the game window looses focus, the action is buffered and still active.
Even after the window gains focus again, and no actual input occurs, the action is still "true".
Steps to reproduce:

open and run the sample project.
press and hold one of the arrow keys -> the label displays which butons are pressed and a counter increases.
click outside of the window so the window looses focus.
--> the counter still increases and the action is still active.

Minimal reproduction project:

action_window_focus.zip

That might be expected behavior though.

It feels somehow wrong.
For example when I have a player character and hold the right arrow key to make him move right and for whatever reason, the window looses focus, then the player character will continue walking right and probably run into an enemy or fall into a pit.
Is there a way to clear the buffer? Or a workaround?
Let's say in "_notification(what)" I can check if the window is off focus and then clear all actions.

This is a common byproduct of focus change, I have seen this behavior in some other games. However I agree that unless keyboard input is retrieved directly (i.e raw input without window manager events), keys should be considered as unpressed if the game looses focus. I can't see a reason why leaking key states  would be wanted.

@pixelriot I haven't tried much with the direct API, because I process the input events and store results in my own controller class.  (Because I need to do some extra work to smooth things out.)
While having things stored there, I have been able to use the notifications from https://docs.godotengine.org/en/3.1/classes/class_mainloop.html
NOTIFICATION_WM_FOCUS_IN
NOTIFICATION_WM_FOCUS_OUT
I use these to clear my controls.  It so far is working fine, but off the top of my head I'm couldn't suggest how to tailor an approach using logic more directly stemming out of _input().
There has also been work recently with this pending pull request:  https://github.com/godotengine/godot/pull/28061
Haven't tested it yet myself, so I don't know exactly what it does.  But if you do your own builds you might want to cherry-pick it and see if it works for your situation.

Thanks @avencherus,
this worked for me (maybe it helps someone else):
 func _notification(what):
	if what == MainLoop.NOTIFICATION_WM_FOCUS_IN:
		for ac in ["ui_left", "ui_right", "ui_up", "ui_down"]:
			Input.action_release(ac) #clear input
	elif what == MainLoop.NOTIFICATION_WM_FOCUS_OUT:
		if !get_tree().is_paused(): #virtual press pause button
			var new_action = InputEventAction.new()
			new_action.pressed = true
			new_action.action = "ui_cancel"
			get_tree().input_event(new_action)

That might be expected behavior though.

https://github.com/godotengine/godot/pull/30792#issuecomment-514587093
This indeed could be a problem but I haven't reached that point yet to verify that, so just referencing this here for now.


Add many customizable params

Added these customizable params (related to https://github.com/Quilljou/transmart/issues/4)

Name
Type
Description
Required

modelContextLimit
number
Optional max context window that the model supports. For example for gpt-4-32k, the context is 32768 tokens. Default to 4096 (gpt-3.5-turbo)
No

modelContextSplit
number
Optional ratio to split between number of input / output tokens. For example, if the input language is English and output is Spanish, you may expect 1 input token to produce 2 output tokens. In this case, the variable is set to 1/2. By default, modelContextSplit is set to 1/1
No

systemPromptTemplate
function
(For advanced usage) Custom prompt template. See "translate.ts" for the default prompt.
No

additionalReqBodyParams
any
(For advanced usage) Custom parameters to be passed into request body. Useful if you use a self-hosted model and you want to customize model parameters. For example, see llama.cpp server example
No

Also added a function to filter out the JSON inside the response text (when using GPT-4-preview - the one with 128k context, sometimes the model add some texts before/after the JSON)
const findValidJSONInsideBody = (input: string): string => {
  const firstBracket = input.indexOf('{')
  const lastBracket = input.lastIndexOf('}') + 1
  return input.substring(firstBracket, lastBracket)
}

Thanks a lot @ngxson

Published to npm @0.3.0


build with python 2.7 in the correct python site dir

Right now the python bindings are not being installed in the correct site dir.  It tries to make a guess which may not be correct depending on the distro.  This patch fixes that by getting the site dir from the python interpreter.
This also hardcodes mraa to build against python 2.7.  It's possible to override this, but currently python3 interpreter detection is broken in cmake < 3.1.

Im not entirley sure about running the python interpreter to get the site dir but if it works everywhere we already build mraa, there shouldnt be a reason why not.
In regards to the patch in itself, please add a sign-off. See https://github.com/intel-iot-devkit/mraa/blob/master/docs/contributing.md

It will add a dependency on python... but it should work.

This is quite cool but I'm not sure I like this method over PR #120. True that #120 means a dep on python setuptools. The problem with either of these methods is when cross compiling and using the cmake IPK package build scenario. I'm tempted to say #120 should be default and when BUILDIPK is set we just do the old behaviour. Let me have a think and get back to you on this.

I didn't know PR #120 existed.  Having a setup.py seems like a typical way to install python packages, however, I'm not sure how standard that is in mixed projects with build tools like cmake...  My PR comes from findings on what other cmake projects have done to install python bindings.

Any additional thoughts on this?  I have other patches in the work I don't want to mix with this if this will be abandoned.

Well honestly your method breaks cross compiling and the other method breaks BUILDIPK, so I'm still working on a solution for everyone :) It's likely I'll take something like your method with a bunch of other bits to do both python2 & 3 builds that I've been working on but haven't quite had time to polish with other stuff coming up - hopefully this week I get a chance to push all that stuff out.
If you have other patches in the pipeline I recommend you send them from different branches as long as they don't need to go on top....

Merged - with a few modifications so that it works in Arch Linux & Poky (YP) as well as debian


Fix return type of textDocument/prepareCallHierarchy

Thanks for open-sourcing this library! It's quite handy. 🙇  While working with it, I discovered a small issue.
The calls for preparing the call hierachy were returning a wrong type. They were returning the response for the prepare rename call.
Additionaly, these parts of the code were using a different indentation compared to the rest of the file.
Is there an easy way how I can add tests for it?

Hello! Thanks so much for the contribution.
First, about the indentation changes. I received feedback a while ago that tabs are a significant accessibility improvement. So, I began migrating to using them. But, I've only been applying the changes as I go. I know this isn't ideal and in fact, might actually be a step down in accessibility. Perhaps it's time finally deal with this correctly.
Second, in general, I haven't been good at producing tests targeting exactly this kind of thing. And, that's something that should definitely be improved. Perhaps we can start by capturing some real data and start here?

Very good points. I've updated the readme with some clarification, along with an .editorconfig file that matches it.
I think expanding testing is a wonderful idea. I have quite a few integration tests within a project that uses this library, but I think making more targeted tests here makes a lot of sense. I'm going to open an issue to cover that.


Extension is currently not working with CTRL + D Shortcut

Hey I've noticed a recent issue with the extension where it is not putting my new bookmarks into the set location when I use CTRL + D. It is however working as expected when I use the Firefox bookmark icon in the address bar (not the extension's quick bookmark icon - I have that option disabled.) The bookmarks are going into the "Bookmarks Menu" when I use the keyboard shortcut, but they are going into the correct folder when I click the blue icon.
This is a recent issue which I've only noticed within the last 24 - 48 hours.

Hello! Thanks for reporting this issue. It seems to have been introduced with the latest version of the add-on. I have been able to reproduce the bug and I am going to work on a fix!

It appears the default bookmark folder is different for the bookmarking shortcut (Ctrl+D) , as well as for the context menu bookmarking on the stable version of Firefox only (57.x). Commit 83dcaa0 fixes this problem. 😃
I have also published version 2.1.1 on AMO , which includes the bugfix. Please do not hesitate to let me know if you are still experiencing the issue after the update!


Create a seperate package for the documentation only

To better maintain the linux packages the html documentation should be seperated from the main package.
Include the examples in the doc package and move them to /usr/share/doc/erwiz/examples

I decided to leave the documentation as is in the same package. It doesn't make sense to increase the complexity.


Added link for "Contribute to this Project" with Github icon

i added a footer so that if you click on either icon or text it will open your repository in a new tab
Also can please label my Pull Request(PR)         "hacktoberfest-accepted"

Hi @PrasannaBH thanks for your contribution, it would be better if you replaced the external cdn with a svg image inside html

I made changes acccording to you Please check it
And Please label my Pull Request(PR) "hacktoberfest-accepted" i need a T-Shirt

Thanks @PrasannaBH 👍


Feedback about new API for CoAP Observe.

This issue will be used to discuss / provide feedback of new API for CoAP Observe.
This was triggered by https://github.com/open-coap/java-coap/issues/27#issuecomment-1589901519 :

It would be great if you could have a look at least at usage diff: https://github.com/open-coap/java-coap/tree/observation-handlers#coap-client-complete-usage, and give feedback, especially about naming (those are always the hardest ;))

I have regularly same naming problem with Leshan :sweat_smile:
@szysas if I want to test that version :point_up:. What is the recommended way ?

is there a nightly build available in a maven repo ?
OR should I build locally ?

Created a PR: https://github.com/open-coap/java-coap/pull/57

About InboundSubscriptionManager API,  I understand that a java-coap user need  to call InboundSubscriptionManager. sendObservation(String uriPath, CoapResponse obs). It seems that this class also hold "observe relation" internally.
So I understand that each time one resource of my CoAP resources tree is modified, I need to generate a CoapResponse  to pass to subMgr.sendObservation(...); and this will be up to subMgr to know if this should be used to send Notification to Observers.
If I'm correct, I think there is a little issue with this design, ideally a CoapResponse  should be created only if notification need to be send.
There is maybe another issue with following use case :
Imagine a resource /test and 2 different observer A and B.  Both want to observe that resource but one uses a GET request with SenML-JSON  content format and another with SenML-CBOR.

Thanks @sbernard31,
You have really good points about naming, I will do changes:

ObservationsStore (instead of ObservationRelationsStore)
NotificationsReceiver (instead of ObservationListener )
ObserversManager (instead of InboundSubscriptionManager)

If I'm correct, I think there is a little issue with this design, ideally a CoapResponse should be created only if notification need to be send.

The idea behind passing CoapResponse is to pass not only payload but also coap headers (like content-type, etag..etc)
Are you worried about performance with that approach?
What if, we replace CoapResponse with a function Supplier<CoapResponse> to lazily create CoapResponse only if there are any observers?

There is maybe another issue with following use case :
Imagine a resource /test and 2 different observer A and B. Both want to observe that resource but one uses a GET request with SenML-JSON content format and another with SenML-CBOR. I think current design will not work with this 🤔

I'm aware of this limitation, which was also before changes we are discussing. I would suggest to take it as separate issue.

The idea behind passing CoapResponse is to pass not only payload but also coap headers (like content-type, etag..etc)

Yep I understand.

Are you worried about performance with that approach?

Exactly, I can imagine use case where resource tree change a lot and if each modification means encoding in complex content format (like SenML-JSON or CBOR), this will not be so good. And  I'm afraid this will not be compatible with other issue, I reported.

What if, we replace CoapResponse with a function Supplier to lazily create CoapResponse only if there are any observers?

Maybe we can even replace it by a   Service<CoapRequest, CoapResponse> instead of Supplier<CoapResponse>
That means that you need to store the original CoapRequest used to register the observer.
That also means that if wanted you can plug the RouterService (or composed service based on it) to retrieve the CoapResponse.
(This will looks like a bit what I did but not in a really elegant way at : https://github.com/eclipse/leshan/commit/10ab042ac56cd5757cfc9814d6af8b86dd391489)
This solution could also solve the other issue I reported

I'm aware of this limitation, which was also before changes we are discussing. I would suggest to take it as separate issue.

We can do that but I'm afraid that I mixed both problem :grin:  just above.

Maybe we can even replace it by a  Service<CoapRequest, CoapResponse> instead of Supplier
That means that you need to store the original CoapRequest used to register the observer.
That also means that if wanted you can plug the RouterService (or composed service based on it) to retrieve the CoapResponse.

That is a neat idea, I like it :)

Note, I updated branch with above changes (https://github.com/open-coap/java-coap/tree/observation-handlers#coap-client-complete-usage)

Good point and good idea.
I will try first with Predicate<String> resourceFilter.

I adapted my code to new API. If you are curious about what it could looks like :
At Client side  : https://github.com/eclipse/leshan/commit/72104597a6d27683b31dd9cf098f9a09289f5d61
At Server side : https://github.com/eclipse/leshan/commit/bb1243070c1773a63b162442eca218e6c34c3e74
This is clearly better.
At client side :
For now, I have 1 concern : How to handle the case where we try to add the Observation to the ObservationStore but there is already an Observation registered for this "identifier" ?
In Californium the code looks like that, this is done before we send the request. But I don't know if I like this way because this means that if something goes wrong we need to remove it from the store.
There was a long discussion about that  long time ago at Californium repository : https://github.com/eclipse-californium/californium/issues/500. (That was so long time ago that I didn't remember everything about this :sweat_smile: )
For now, I don't know what could be the best move. I share this just to let you know.
I think this is maybe too soon to address this. Knowing that for now, I didn't explore to use java-coap with a secure transport.
At Server side:
I'm not able to implement LWM2M Composite Observe feature.  This feature allows to observe several resources with only 1 FETCH (not GET) request on "/" (resources to observe are define in the payload.
(If you ask yourself like me if this is possible to have several relation from same peer on "/", you could have a look at : https://github.com/OpenMobileAlliance/OMA_LwM2M_for_Developers/issues/528)
Current code does support that :point_up: ,  I think I will also need   Predicate<CoapRequest> ObserverFilter because I need the CoapRequest payload which contains the resources to observe (or ideally the decoded payload which I could attached to CoapRequest ? I don't know if I will be able to do that ? )
Let me know if you need more clarification for this 2 points :point_up:

How to handle the case where we try to add the Observation to the ObservationStore but there is already an Observation registered for this "identifier" ?

Overwrite?
Anyway, there should be enough identifiers in CoapRequest to create unique observation identity per device.

At Server side

I see your point but I am not really sure how changing into Predicate<CoapRequest> helps.
One way to handle composite observation would be to create Filter like:
        Filter.SimpleFilter<CoapRequest, CoapResponse> compositeSubscriber = (request, service) -> {
            if (request.getMethod() == Method.FETCH){
                // - parse payload
                // - create multiple observation requests
                // - call underlying service multiple times
                return completedFuture(CoapResponse.ok(":)"));
            } else {
                return service.apply(request);
            }
        };

Then compose it with ObserversManager:
compositeSubscriber.andThen(observersManager).then(rootResource)

Overwrite?
Anyway, there should be enough identifiers in CoapRequest to create unique observation identity per device.

Yep, ideally the  observation should be identified by "device identity + token". But currently (for californium historical reason), Leshan are using token only.
Anyway, the point is more that It's hard to be sure that Token used for "observe request" will not be already used when ObservationsStore.add(CoapRequest)
Even if,  I check if token is not already used (at cost of 1 more call to shared stored) when I generate it, I can not be sure it will not be used at "add" time. (e.g. because of race condition)
I just let you know the problem, just in case, you have an idea to handle it.
On my side, I will think more about it. And to be more clear,  I think current design at client side is a good first step.

About compositeSubscriber
Maybe I don't get you but I think it will not work. :thinking:
The CompositeObserve is a way to observe several resources at same time but when only 1 resource changes then all the observed resources are sent in notification.  (This is like if you create a kind of new virtual resource)
CompositeObserve is a LWM2M feature and so must not be part of java-coap but using FETCH for observe is a CoAP one.  (See https://www.rfc-editor.org/rfc/rfc8132.html#section-2.4) So ideally we should find an API which support FETCH for CoAP (which is very general feature) and that API should be usable for specific LWM2M Composite Observe feature.
About FETCH, the RFC7252§1.1. FETCH says :

The CoAP GET method [RFC7252] is used to obtain the representation of
a resource, where the resource is specified by a URI and additional
request parameters can also shape the representation.
...
As an alternative to using GET, many implementations make use of the
POST method to perform extended requests (even if they are
semantically idempotent, safe, and even cacheable) to be able to pass
along the input parameters within the request payload as opposed to
using the request URI.
As with POST, the input to the FETCH operation
is passed along within the payload of the request rather than as part
of the request URI.  Unlike POST, however, the semantics of the FETCH
method are more specifically defined.

So I feel that using URI as main identifier in ObserversManager  is not enough if you want to support FETCH, It seems that you need payload too. So maybe the right level is CoapRequest which hold URI + payload ?
or I missed something ?

I see your point but I am not really sure how changing into Predicate helps.

The idea is to implement something like this  :
https://github.com/eclipse/leshan/blob/bb1243070c1773a63b162442eca218e6c34c3e74/leshan-tl-javacoap-client/src/main/java/org/eclipse/leshan/transport/javacoap/client/observe/NotificationHandler.java#L38-L48
But for Observe Composite it will looks like :
   /** Called when resources changed */
   public void resourceChanged(LwM2mPath... paths) {
        // send notification if there is observers for it
        observersManager.sendObservation((coapRequest) -> {
            List<LwM2mPath> listOfResourceObservedByCompositeObserve = getPaths(coapRequest);
            if (listOfResourceObservedByCompositeObserve != null) {
            for (LwM2mPath observePath : listOfResourceObservedByCompositeObserve) {
                for (LwM2mPath path : paths) {
                    if (path.startWith(observePath)) {
                        return true;
                    }
                }
            }
        }
        }, responseProvider);

    /** get LWM2M path to observe from coap request
    public List<LwM2mPath> getPaths(CoapRequest coapRequest) {
     // here I can decode payload which contains LWM2M path for each call
     // but ideally I would prefer to be able to decode it once and then attach it to the request.
     }

Stupid idea but what if you simply make a counter and increment each time subscription is send? With this approach you have extremely small chances that token is reused (hard to believe that there will be 281 474 976 710 655 subscriptions)

Thx for the idea :pray: , maybe this is way to explore :thinking:
This counter should be in a shared store and should be called each time we need to send a new observe request. (Token used should not conflict with none observe request, so maybe we should have 2 exclusive subset of token : one in shared store for observe another in memory for none observe request)

About composite observe:
I agree that it should not be part of this coap library, that's why we should not add any special functionality into ObserversManager (I don't think it would help anything). In other words it should not be aware of any other way of subscribing (like FETCH request).
My suggestion is to use API of ObserversManager to add subscriptions (observation relations). We could have simpler API like:
observersManager. subscribe(token, uriPath, peerAddress)
Then, when doing usual:
observersManager.sendObservation(uriPath, ...)
Works the same as with "normal" subscriptions.
I created test use case that shows this: https://github.com/open-coap/java-coap/blob/observation-handlers/coap-core/src/test/java/com/mbed/coap/server/observe/ObserversManagerTest.java#L205
Another alternative is to implement own ObserversManager in Leshan, that is specific to LwM2M. Note that ObserversManager in java-coap is not coupled to anything so it can be easily replaced.
What do you think?

Looking at your previous comment and also code your share,  it seems to me there is 2 points about Observe with FETCH we don't understand in a same way. I think this is important that we agree on this (or at least agree on disagreement :sweat_smile:) before to enter in implementation discussion :slightly_smiling_face:
So let's focus on CoAP only (ignoring LWM2M for now).
I try to clarify my understanding :

Observe with FETCH is part of CoAP  via RFC8132. So ideally java-coap should support it by providing an API to help user to implement observe feature based on FETCH. So it makes sense to first try to find that API  (but I agree that if we didn't find anything useful, a fallback could be to let user implement their own ObserversManager)

Observe with FETCH is not about creating several observe relation with one request, it is about creating one observe relation based on "URI + payload" instead of just the "URI". If you look at RFC8132§2.7.  A Simple Example for FETCH, you will see another usage than the LWM2M Composite Operation one. In that case the FETCH is used to request subpart of the resource. With observe this will be about observing a subpart of the resource. To know what is observed you need the URI  + the payload.

Let me know, if there is points you disagree with :slightly_smiling_face:

Partially agree.

So let's try to clarify point 2. together.

With GET, you can subscribe for different content type (accept) as well as with FETCH but in addition there is a payload. That payload will be used when creating notification same as accept header.

I agree that accept + payload is needed to create the "notification".
But I think it is also needed to identify if a  "change" of the observed resource need to trigger a notification or not.
Still with RFC8132§2.7. A Simple Example for FETCH, you can observe :
   FETCH CoAP://www.example.com/object
   Content-Format: NNN (application/example-map-keys+json)
   Accept: application/json
   [
     "foo"
   ]

and also :
   FETCH CoAP://www.example.com/object
   Content-Format: NNN (application/example-map-keys+json)
   Accept: application/json
   [
     "x-coord"
   ]

Both observation relation target CoAP://www.example.com/object URI but one should be triggered with x-coord change and the other when foo change.
(About both relation on same URI, I share again this post : https://github.com/OpenMobileAlliance/OMA_LwM2M_for_Developers/issues/528 which is an OMA/LWM2M question but solve with a CoAP answer from IETF)

Going back to ObserversManager API, it already has a callback in sendObservation method, that helps with constructing proper notification.

I let aside implementation question for now.

But I think it is also needed to identify if a "change" of the observed resource need to trigger a notification or not

OK, I see your point, so in addition to different notifications content, we might have no notification at all.
Agree with you on this ;)

Agree with you on this ;)

Ok so I think we can go back to code discussion. :slightly_smiling_face:
I understand the ObserversManager hold all "observe relation" at serve side.  (It manages registering / unregistering)
And User of ObserversManager is responsible to  :

know when something "change" in the system.
know if a given relation is concerned by a "change" from  To be Defined ?
Generate the "notification"/"coap response" from CoAP request used to register the observation.

I think the main missing point is 2, that's why I proposed the  Predicate<CoapRequest> ObserverFilter which should work ?
But my concern with that solution is that : coaprequest contains the payload which is needed to know "if relation is concerned by a change" but this information is in the payload and need to be decode each time. It would be better to have a way to attach the decoded data to something ? Maybe we need an ObserveRelation object, I don't know ?

Uff ;)
Adding Predicate<CoapRequest> observerFilter would cause that we need to iterate through every subscriber of any resource, I'm not very sure about it.

But my concern with that solution is that : coaprequest contains the payload which is needed to know "if relation is concerned by a change" but this information is in the payload and need to be decode each time. It would be better to have a way to attach the decoded data to something ? Maybe we need an ObserveRelation object, I don't know ?

And here we are entering area that is becoming more LwM2M specific. So maybe implementing ObserversManager in Leshan, with all optimisations related to composite observation could be an option?

Adding Predicate observerFilter would cause that we need to iterate through every subscriber of any resource, I'm not very sure about it.

Yep, I understand the concern. Note that this is also true for Predicate<String> uriPathFilter
At the same time, I'm ask myself if having a lot of observe relation at same time is a real use case, so maybe iterate is not really an issue :thinking:

And here we are entering area that is becoming more LwM2M specific.

I disagree, I think often when you will use FETCH you will have information in the payload which need to be decoded.
Some  examples:

SenML  : https://www.rfc-editor.org/rfc/rfc8790.html#section-3.1 which is not specific to LWM2M
DNS over CoAP : https://www.ietf.org/archive/id/draft-ietf-core-dns-over-coap-02.html#section-4.2

So maybe implementing ObserversManager in Leshan, with all optimisations related to composite observation could be an option?

Yep this is clearly an option but I feel too bad that we don't find something at java-coap level because I feel there is some part of the logic which should be sharable.
If you agree, I will try to experiment this a little and maybe come back to you with some code. Of course, feel free to reject the codes/ideas if you think that this will not go in the right direction for java-coap. :slightly_smiling_face:
Anyway thanks for this constructive discussion, this helps me to clarify my mind about all of this. :pray:

Yep, I understand the concern. Note that this is also true for Predicate uriPathFilter
At the same time, I'm ask myself if having a lot of observe relation at same time is a real use case, so maybe iterate is not really an issue

Well, there will less subscribed uri-paths, and comparing just Strings is a lot easier.. but that just optimisation.

If you agree, I will try to experiment this a little and maybe come back to you with some code. Of course, feel free to reject the codes/ideas if you think that this will not go in the right direction for java-coap. 🙂

👍

Here is my first try : https://github.com/eclipse/leshan/commit/27393771023e61909a518f828c9eec775c9c0449
This is a draft.
ObserversManager, ObserverStore and HashMapObserversStore is pure CoAP class.
The rest is how I used it in Leshan.
The store allows :

to define custom key for Observers (by default  I use token + peeraddress)
to attach UserData.
to persist relation if needed (I don't use it but I know by the past some CoAP users wanted to do that)

We still have the "iterate over all observers issue" but I don't know if this is a real issue or if fixing it will be a premature optimization but If you want me to try to find a solution for it, I can try.
Let me know, what you think about that and if we should go deeper in that way.

@sbernard31
I will come back to this in August, after my summer vacation which just started ;).

Hi @sbernard31 , should we come back to this discussion after long break?
I still have a concert if using ObserverManager for composite observations is a right way forward for now, but in general observation mechanism from CoAP and LwM2M is not the simplest and so I'm probably missing the point.
Is the ObserverManager functionality the only issue right now? If so, could we merge it (PR: #36) to master and keep working on it separately?

Hi @sbernard31 , should we come back to this discussion after long break?

Just came back.

I still have a concert if using ObserverManager for composite observations is a right way forward for now

I understand.

Is the ObserverManager functionality the only issue right now? If so, could we merge it (PR: https://github.com/open-coap/java-coap/issues/36) to master and keep working on it separately?

Yep let's do that.


Honor a hostedZone construct if given as props

Seems the linter made a few extra formatting changes.  The core change is in lookupHostedZone.
Fixes #614

Maybe missing something but doesn't this construct already fall back to this functionality here: https://github.com/serverless-stack/serverless-stack/pull/622/files#diff-2cadf789af37475cf8e39f004b698c903e93773b4740b82afecbd48f82122bcaR342

Got it! Makes sense

@mtdewers can you rebase this against master? We just ran prettier across the codebase

rebase complete

Now that's a nice looking diff :smile: Will get this out in the next release


tf2-automatic crashed issue: Cannot read 'length' property

09-04-2019 01:03:25 - error: tf2-automatic crashed! Please create an issue with the following log:
crash: Automatic.version: 1.12.1; node: v10.15.2 win32 ia32; Contact:<PHONE_NUMBER>3911511
crash: Stack trace:
TypeError: Cannot read property 'length' of undefined
at Schema.getName (C:\Users\Izzuddin\Documents\bot1_SmileBot\node_modules\tf2-items\lib\schema.js:230:40)
at getName (C:\Users\Izzuddin\Documents\bot1_SmileBot\app\items.js:132:25)
at Object.createDictionary (C:\Users\Izzuddin\Documents\bot1_SmileBot\app\items.js:58:20)
at C:\Users\Izzuddin\Documents\bot1_SmileBot\app\inventory.js:104:34
at C:\Users\Izzuddin\Documents\bot1_SmileBot\app\inventory.js:63:9
at SteamCommunity. (C:\Users\Izzuddin\Documents\bot1_SmileBot\node_modules\steamcommunity\components\users.js:484:5)
at Request._callback (C:\Users\Izzuddin\Documents\bot1_SmileBot\node_modules\steamcommunity\components\http.js:67:15)
at Request.self.callback (C:\Users\Izzuddin\Documents\bot1_SmileBot\node_modules\steamcommunity\node_modules\request\request.js:185:22)
at Request.emit (events.js:189:13)
at Request. (C:\Users\Izzuddin\Documents\bot1_SmileBot\node_modules\steamcommunity\node_modules\request\request.js:1161:10)
09-04-2019 01:03:25 - error: Create an issue here: https://github.com/Nicklason/tf2-automatic/issues/new
Press any key to continue . . .

Delete the schema.json file from the temp folder and restart the bot, that will fix it.

Thank you @Nicklason , I've deleted schema.json file and the error is gone, but new error just appeared:
09-04-2019 01:13:31 - error: An error occurred while initializing the packages: tf2-items (ETIMEDOUT).
Press any key to continue . . .
Please help.. I don't know why this happened.

That's a temporary error, just try running it again

Thank you! Problem solved :)


Allow caching of compiler version and argument information to improve startup time

add something like the following to compiler-explorer.local.properties
compilerCacheConfig=OnDisk(./lib/storage/data/cache, 50)

Startup when the cache is empty is marginally slower, but subsequent startups can be an order of magnitude faster, because no compilers will be executed during startup.
With 70 compilers installed locally the reported startup time went from 11604ms to 1711ms.
With 3 compilers installed locally the reported startup time went from 1742ms to 1429ms.
Definitely useful during development, although in production the usefulness really depends on how fast S3 caching would be.  It may simply be faster to execute the compilers off EFS than to make several hundred round trips to S3.

This looks great: thanks. I'd bet the round-trip to S3 is still faster than running the compiler. We lose the "precaching" of the compilers in RAM, but I bet we run out of free cache RAM before we finish running all the compilers anyway...

All merged...eventually. Sorry for the trouble...


feature request- update_message/2 (passing in metadata as well as data)

I've got a situation where i'm trying to update a message based on some rabbitmq metadata.
The definition of Message is this;
  @spec update_data(message :: Message.t(), fun :: (term -> term)) :: Message.t()
  def update_data(%Message{} = message, fun) when is_function(fun, 1) do
    %Message{message | data: fun.(message.data)}
  end

So the normal handling looks like
message
 |> Message.update_data(&process_data/1)
 |> Message.put_batcher(:elasticsearch)

This means, my process data method doesn't get the full message, just the data field.  I really also want the metadata field, because I want to pull out the rabbitmq exchange, routing key and correlation id when i'm processing.
so I'm thinking the best way would be to add a new method, something like
  @spec update_data(message :: Message.t(), fun :: (term -> term)) :: Message.t()
  def update_data(%Message{} = message, fun) when is_function(fun, 2) do
    %Message{message | data: fun.(message.data, message.metadata)}
  end

thoughts?

In this case, you can just access message.metadata inside the update_data
block.
José Valim
www.plataformatec.com.br
Skype: jv.ptec
Founder and Director of R&D


Not getting match.

Hello,
I have set up elastalert on elasticsearch 6.2.2
I am using any type with the following configuration:
es_host: localhost
es_port: 9200
_source_enabled: true
name: Log checker for SEVERE in Jetty
type: any
index: filebeat-*
timeframe:
  minutes: 5
filter:
- term:
    system.jetty.severity: "SEVERE"
aggregation:
  minutes: 5
aggregation_key: '@environment'
alert:
- "email"
- "slack"
slack_webhook_url: webhook url
slack_username_override: VC-alerts
slack_msg_color: 'danger'
email:
-<EMAIL_ADDRESS>
running elastalert-test-rules does not give any match however running the following query in elasticsearch I get the some results:
curl -XGET "http://localhost:9200/filebeat-*/_search" -H 'Content-Type: application/json' -d'
{
  "query": {
    "match": {
      "system.jetty.severity": {
        "query": "SEVERE"
      }
    }
  }
}'

Any idea?

Any idea?

I think you have same trouble https://github.com/Yelp/elastalert/issues/1449

Nope. even not running in test mode I don't get any match which is not correct because I should get lots of matches in 15 minutes timestamp.
I guess that the issue is coming from not having python elasticsearch module for version 6.2.2. Who knows when that will be released.

Hi there!
I have the same issue, not getting matches for all rules using Elasticsearch 6.2.2.
Any idea what is the issue here?
Thanks in advance.


feat(refactoring): k2s PowerShell codebase into modules and scripts

Motivation

Scattered scripts across codebase to setup a cluster is daunting and arduous to setup a variant of k2s.
Need for structured modules which can be reused to build variant of k2s.

Following tasks in no order needs to complete:

[x] Create base variant of k2s.
[x] Create multi-vm variant of k2s.
[x] Create dev variant of k2s.

Base installation of all three major variants are complete, next steps shall be taken up in a new issue.


Refactor Dataframe-related classes

This PR includes several refactors:

[x] Refactor inside Dataframe class into smaller, more meaningful functions (and removing some duplications)
[x] New dataframe namespace, for Dataframe-related classes
[x] Extract new ViewportFeature class generator from Dataframe
[x] Extract new 'helper classes' from Dataframe:

[x] FeatureIdsHelper
[x] ProjectionHelper
[x] ViewportHelper

Hey @Jesus89 would you mind having a look at this one?

Sure!

Are you ok with merging this @elenatorro ?


add nacos config paramter by context-path

Why you need it?
Is your feature request related to a problem? Please describe in details
nacos 服务context-path非默认的情况下，不支持参数的指定
How it could be?
A clear and concise description of what you want to happen. You can explain more about input of the feature, and output of it.
Other related information
Add any other context or screenshots about the feature request here.
建议在Nacos配置参数中增加context-path支持

@llm163520 any nacos's doc link about this issue?

https://github.com/alibaba/nacos/blob/master/client/src/main/resources/nacos_default_setting.properties

PR #5111 已合并


how to generate new raw lut file in different color-style

how to generate new raw lut png file in different color-style  except existed file such as 0.original.png  1.Clarendon.jpg 3.Moon.jpg 15.Hudson.jpg
this reverse engineering  shows me how to use the existed filter but doesn't show how to generate filter from image

Have you tried the guide? https://github.com/homm/color-filters-reconstruction#guide


SKF usability

Context
LNS hosts may set up many routes all sharing the same DevAddr ranges. For instance, they could set up one route for each organization/tenant/user of their LNS, that way they can configure a max_copy setting on an organization/tenant/user level.
The current SKF works as following
• If a route has no SK on one of its DevAddr, then HPR will send all packets under that DevAddr to the route
• If a route has one or more SK on one of its DevAddr, then HPR will only send packets under that DevAddr if and only if HPR can validate the MIC using one of the SKs
Let's say an LNS host has 1000 organization/tenant/user, and 1000 DevAddr, neither of which are wildly unrealistic.
It is very much possible that each organization/tenant/user may not have an active device occupying each of the 1000 DevAddr, let's call those "inactive DevAddr." Note that one route's inactive DevAddr may be active in another route.
In order to avoid getting bombarded by traffic on inactive DevAddrs, LNS hosts may submit dummy keys on them, so HPR would fail at validating the MIC.
This means that the LNS host could be submitting 1000*1000 = 1mm SKs, despite the fact that many of them may simply be dummy keys.
This puts unnecessary burden on HPRs and config service.
Proposal
Make a route-level flag "get all traffic for that route's DevAddr ranges"
• If the flag = true, then HPR would ignore all SKF on that route
• If the flag = false.
• When there is no SK on a DevAddr, instead of defaulting to "get all traffic for that DevAddr,"  default to "block all traffic for that DevAddr." That way, LNS hosts don't need to submit unnecessary dummy keys for inactive DevAddrs.
• If a route has one or more SK on one of its DevAddr, then HPR will only send packets under that DevAddr if and only if HPR can validate the MIC using one of the SKs [unchanged from previous behavior]

Also need to think about how we facilitate the change to the new SKF scheme so we don't have a breaking change.

Revised Proposal
Make a route-level flag "ignore empty skf"
If the flag = true
• When there is no SK on a DevAddr, HPR blocks all traffic under that DevAddr to the route. That way, LNS hosts don't need to submit unnecessary dummy keys for inactive DevAddrs.
If the flag = false
• When there is no SK on a DevAddr, HPR sends all traffic under that DevAddr to the route.
Regardless of the flag, when there is an non-empty SKF, then HPR will respect it (i.e. send only if the traffic can be validated by one of the SKs in the non-empty SKF)


Wrong documentation link in README

Hi!
Thanks for the library! I guess now it is kinda proven that userver really exists :)
Regarding the problem. In README you have a link to the documentation: See the docs for more info. Please change the link to the actual documentation which is public accessible.
Thanks in advance!

Many thanks for the bug report! Fixed in https://github.com/userver-framework/userver/commit/8a3276d18a037459a639b6ebb6af0e9a7ed4f35c


Error shapes not alligned when creating Instance with numpy array

It is quite common in data science apps to use numpy arrays and pandas dataframes. Unfortunately when you try to create Instance using numpy array of shape (1,x) your code creates in one of the places array of shape (nobs, x, 1) which does not work in dot when multiplied by array of shape (x, nlayers). It could be easily checked with the verify_dataset_shape_and_modify. Here's some snippet:
def mult(x):
     res = x[0]
     for a in x[1:]:
         res *= a
     return res
[...]
def verify_dataset_shape_and_modify(...)
    [...]
    data              = np.array( [instance.features.reshape((mult(instance.features.shape),) \
                                         if type(instance.features) == np.ndarray\ #more generalization perhaps?
                                         else instance.features\
                                         for instance in dataset ] )

whith this snippet even accidentally passing arrays of shape (x,1) will work.

You are correct. The documentation states that an instance should be initialized with a built-in one-dimensional python list, e.g. Instance( [0, 1] ). However, as you pint out there are currently no checks that verify whether the argument is indeed a python list.
I'll add validation check to my todo list, thank you for pointing it out


[py2py3] modernize scripts that rely on urllib

Fixes #10481
Status
ready

dario tested validate-test-wfs with py3 env
over slack, Alan confirmed he validated both scripts both in py2 and py3.

Description
Modernize the scripts that have been left behind in #10482 , by leveraging the same "hacky" technique used in #9934
Is it backward compatible (if not, which system it affects?)
yes
Related PRs
follow up to #10482
External dependencies / deployment changes
nope

Thanks for providing these changes, Dario.
I have tested those in a python3 environment/agent, and here is the outcome for workflowCompletion.py (working fine!):
cmst1@vocms0192:/data/srv/wmagent/current $ python3 apps/wmagentpy3/bin/adhoc-scripts/workflowCompletion.py -w amaltaro_TaskChain_PUMCRecyc_Agent152_Val_210817_124215_6631 -c cmsweb-testbed.cern.ch -r alancc7-cloud1.cern.ch
apps/wmagentpy3/bin/adhoc-scripts/workflowCompletion.py:58: DeprecationWarning: key_file, cert_file and check_hostname are deprecated, use a custom context instead.
  return HTTPSConnection(host, key_file=self.key, cert_file=self.cert)
==> amaltaro_TaskChain_PUMCRecyc_Agent152_Val_210817_124215_6631	(status: completed)
InputDataset:
    /RelValQCD_Pt_600_800_14/CMSSW_11_2_0_pre8-112X_mcRun3_2024_realistic_v10_forTrk-v1/GEN-SIM (total input lumis: 180)
OutputDatasets:
    /RelValQCD_Pt_600_800_14/CMSSW_11_2_0_pre8-DigiPU_2024PU_TaskChain_PUMCRecyc_Agent152_Val_Alanv11-v11/GEN-SIM-DIGI-RAW (lumis: 180, lumi completion: 1.0)
    /RelValQCD_Pt_600_800_14/CMSSW_11_2_0_pre8-RecoPU_2024PU_TaskChain_PUMCRecyc_Agent152_Val_Alanv11-v11/DQMIO (lumis: 180, lumi completion: 1.0)
    /RelValQCD_Pt_600_800_14/CMSSW_11_2_0_pre8-RecoPU_2024PU_TaskChain_PUMCRecyc_Agent152_Val_Alanv11-v11/MINIAODSIM (lumis: 180, lumi completion: 1.0)
    /RelValQCD_Pt_600_800_14/CMSSW_11_2_0_pre8-RecoPU_2024PU_TaskChain_PUMCRecyc_Agent152_Val_Alanv11-v11/GEN-SIM-RECO (lumis: 180, lumi completion: 1.0)

while we have problems with the validate script:
amaltaro@vocms0192:/data/amaltaro/July2020_Val $ python3 --version
Python 3.8.2
amaltaro@vocms0192:/data/amaltaro/July2020_Val $ python3 validate-test-wfs.py -w amaltaro_TaskChain_PUMCRecyc_Agent152_Val_210817_124215_6631 -c cmsweb-testbed.cern.ch -r alancc7-cloud1.cern.ch
validate-test-wfs.py:68: DeprecationWarning: key_file, cert_file and check_hostname are deprecated, use a custom context instead.
  return HTTPSConnection(host, key_file=self.key, cert_file=self.cert)
Traceback (most recent call last):
  File "validate-test-wfs.py", line 713, in <module>
    sys.exit(main())
  File "validate-test-wfs.py", line 673, in main
    rucioToken = getRucioToken(rucioUrl)
  File "validate-test-wfs.py", line 132, in getRucioToken
    token = getContent(rucioAuth, headers=[("X-Rucio-Account", rucioAcct)])
  File "validate-test-wfs.py", line 100, in getContent
    output = response.headers.getheader('X-Rucio-Auth-Token')
AttributeError: 'HTTPMessage' object has no attribute 'getheader'

Ugh, you are correct! i saw something similar with DQMUpload but forgot about it.
I added a new commit that has two options:

            output = response.headers.get('X-Rucio-Auth-Token', '')  # option 1 is similar to what we are doing in DQMUpload at https://github.com/dmwm/WMCore/blob/88b70a21208c1b4b849a122336c600210a71c487/src/python/WMCore/WMSpec/Steps/Executors/DQMUpload.py#L292
            # output = response.getheader('X-Rucio-Auth-Token')  # option 2 is what is suggested but the python3 doc, but this is likely not compatible with py2.

Can i ask you @amaltaro to test if one of these two options work? Thank you!

@mapellidario the current implementation you have in this PR is now working. However, it now fails down the road in:
----------------------------------------------------
==> amaltaro_TaskChain_PUMCRecyc_Agent152_Val_210817_124215_6631
sys.version_info(major=3, minor=8, micro=2, releaselevel='final', serial=0)
----------------------------------------------------
Comments:
Traceback (most recent call last):
  File "validate-test-wfs.py", line 716, in <module>
    sys.exit(main())
  File "validate-test-wfs.py", line 682, in main
    reqmgrInputDset, reqmgrOutDsets = handleReqMgr(reqName, reqmgrUrl)
  File "validate-test-wfs.py", line 498, in handleReqMgr
    twPrint(reqmgrOut.get('Comments', ''))
  File "validate-test-wfs.py", line 428, in twPrint
    print(twPrinter(obj))
  File "validate-test-wfs.py", line 406, in twEnclosed
    for key, value in obj.iteritems():
AttributeError: 'collections.OrderedDict' object has no attribute 'iteritems'

Oh no! I forgot to run futurize on that script! it should be ok now :grimacing: Sorry for the inconvenience!

And now we have another problem. It's probably better if we discuss it over slack and I show you how to test it. Just ping me when you are available, please.

Thanks for following this up and getting all those issues fixed, Dario.


Component: Native browser selectbox

e.g. https://www.figma.com/community/file/830933629718162411

No use case at the moment


No Heroes Allowed: No Puzzles Either!™ [PCSA00150]

App summary

App name: No Heroes Allowed: No Puzzles Either!™
App serial: PCSA00150
App version: 1.00

Vita3K summary

Version: v0.1.8
Build number: 3250
Commit hash: https://github.com/vita3k/vita3k/commit/6342c205
CPU backend: Dynarmic
GPU backend: OpenGL

Test environment summary
Tested by: space1
OS: Windows 11
CPU: AMD Ryzen 5 3600
GPU: Nvidia 3070
RAM: 32 GB
Issues
Graphic issues like no backgrounds ingame.
Screenshots

Log
PCSA00150 - [No Heroes Allowed  No Puzzles Either!™].log
Recommended labels
Playable
graphic bug

sounds like ingame+ to me


Typo fix

Thanks! I see that rebasing this PR to the developer branch will move unecessary merge commits to the developer. Could you do a new PR from your forked up-to-date developer branch? I could push that directly there but i dont want to steal the PR from you 😄

Sure, I've had a go in #19 I'll close this pull request.


Sequelize.sync() error

The error
Unhandled rejection SequelizeDatabaseError: syntax error at or near "false"
    at Query.formatError (/home/noah/Documents/SquadUp-Server/node_modules/sequelize/lib/dialects/postgres/query.js:437:14)
    at null.<anonymous> (/home/noah/Documents/SquadUp-Server/node_modules/sequelize/lib/dialects/postgres/query.js:112:19)
    at emitOne (events.js:77:13)
    at emit (events.js:169:7)
    at Query.handleError (/home/noah/Documents/SquadUp-Server/node_modules/pg/lib/query.js:108:8)
    at null.<anonymous> (/home/noah/Documents/SquadUp-Server/node_modules/pg/lib/client.js:171:26)
    at emitOne (events.js:77:13)
    at emit (events.js:169:7)
    at Socket.<anonymous> (/home/noah/Documents/SquadUp-Server/node_modules/pg/lib/connection.js:109:12)
    at emitOne (events.js:77:13)
    at Socket.emit (events.js:169:7)
    at readableAddChunk (_stream_readable.js:146:16)
    at Socket.Readable.push (_stream_readable.js:110:10)
    at TCP.onread (net.js:523:20)

is thrown when you add the statement
sequelize.sync()

This statement is supposed to enforce the defined schema onto the database. So something is going wrong in the business logic.

Turns out you have to put the timestamps: false option in a separate config object after the schema obj.


Customize api server port

LMK if you are unable to merge

@brandond yeah I don't have permission to merge this


[SPEC] Integrate granular steps into the withdrawing modal UI

Business logic
Currently, we have a state for GeneratingZk which is responsible for displaying the withdrawing modal.
Change the WithdrawState enum in mixer-withdraw.ts:

GeneratingZk state should change to StartingWithdraw. It will serve the same purpose for displaying the withdraw modal.

Then, augment the WithdrawState enum with the following states:

GeneratingWitness state to enter immediately before the call to generateWitness() in webb-anchor.ts
FetchingKeyFromIPFS state to enter immediately before the call to fetchKeyForEdges() in webb-utils.ts
ProvingWitness state to enter immediately before the call to groth16.prove() in webb-utils.ts
VerifyingProof state to enter immediately before the call to groth16.verify() in webb-utils.ts

In order to emit these events, a reference of the BridgeWithdraw / MixerWithdraw needs to be added to the function signature for webb-anchor.ts function generateZKP() and webb-utils.ts function proofAndVerify().
UI Updates
Update the appropriate status text in the withdraw modal with the new states.

Related to: #395


Extension host crashes when opening php files

Whenever I try to open a php file, the extension host crashes. This is the output in the console:
/C:/Program Files/Microsoft VS Code/resources/app/out/vs/workbench/workbench.main.js:2830    Extension host terminated unexpectedly. Code:<PHONE_NUMBER>  Signal:  null
/C:/Program Files/Microsoft VS Code/resources/app/out/vs/workbench/workbench.main.js:2207  Extension host terminated unexpectedly.
Works fine with html or css files. Please fix this asap. I love this editor and have a deadline coming soon. Looking forward to a quick fix. Also happening with the insiders release.
Cheers,
Swarnava.

Version 1.21.1
Commit 79b44aa704ce542d8ca4a3cc44cfca566e7720f1
Date 2018-03-14T14:46:47.128Z
Shell 1.7.9
Renderer 58.0.3029.110
Node 7.9.0
Architecture x64
Error does not occur when I remove the "php" folder from the "extensions" folder.
Here's the Extension Host Output:
[2018-03-31 19:26:50.830] [exthost1] [info] extension host started
[2018-03-31 19:26:50.871] [exthost1] [info] ExtensionService#_doActivateExtension vscode.emmet {"startup":true,"activationEvent":"*"}
[2018-03-31 19:26:50.871] [exthost1] [info] ExtensionService#loadCommonJSModule c:\Program Files\Microsoft VS Code Insiders\resources\app\extensions\emmet\out\extension
[2018-03-31 19:26:51.105] [exthost1] [info] ExtensionService#_doActivateExtension vscode.git {"startup":true,"activationEvent":"*"}
[2018-03-31 19:26:51.105] [exthost1] [info] ExtensionService#loadCommonJSModule c:\Program Files\Microsoft VS Code Insiders\resources\app\extensions\git\out\main
[2018-03-31 19:26:51.596] [exthost1] [info] ExtensionService#_doActivateExtension vscode.merge-conflict {"startup":true,"activationEvent":"*"}
[2018-03-31 19:26:51.596] [exthost1] [info] ExtensionService#loadCommonJSModule c:\Program Files\Microsoft VS Code Insiders\resources\app\extensions\merge-conflict\out\extension
[2018-03-31 19:26:51.627] [exthost1] [info] ExtensionService#_doActivateExtension HookyQR.beautify {"startup":true,"activationEvent":"*"}
[2018-03-31 19:26:51.627] [exthost1] [info] ExtensionService#loadCommonJSModule C:\Users\steph\.vscode-insiders\extensions\HookyQR.beautify-1.3.0\extension
[2018-03-31 19:26:51.999] [exthost1] [info] ExtensionService#_doActivateExtension formulahendry.auto-close-tag {"startup":true,"activationEvent":"*"}
[2018-03-31 19:26:51.999] [exthost1] [info] ExtensionService#loadCommonJSModule C:\Users\steph\.vscode-insiders\extensions\formulahendry.auto-close-tag-0.5.6\out\src\extension
[2018-03-31 19:26:52.012] [exthost1] [info] eager extensions activated
[2018-03-31 19:26:52.047] [exthost1] [info] ExtensionService#_doActivateExtension vscode.php-language-features {"startup":false,"activationEvent":"onLanguage:php"}
[2018-03-31 19:26:52.047] [exthost1] [info] ExtensionService#loadCommonJSModule c:\Program Files\Microsoft VS Code Insiders\resources\app\extensions\php-language-features\out\phpMain
[2018-03-31 19:26:52.181] [exthost1] [info] ExtensionService#_doActivateExtension ecmel.vscode-html-css {"startup":false,"activationEvent":"onLanguage:php"}
[2018-03-31 19:26:52.181] [exthost1] [info] ExtensionService#loadCommonJSModule C:\Users\steph\.vscode-insiders\extensions\ecmel.vscode-html-css-0.2.0\out\src\extension
The error occurs even after running with code --disable-extensions.
On running with code --disable-extensions --logExtensionHostCommunication this is the output:
/C:/Program Files/Microsoft VS Code/resources/app/out/vs/workbench/workbench.main.js:2831 [Extension → Window][len:    24] Array(24)
/C:/Program Files/Microsoft VS Code/resources/app/out/vs/workbench/workbench.main.js:2831 [Window → Extension][len:    26] Array(26)
/C:/Program Files/Microsoft VS Code/resources/app/out/vs/workbench/workbench.main.js:2831 [Window → Extension][len:     1] Array(1)
/C:/Program Files/Microsoft VS Code/resources/app/out/vs/workbench/workbench.main.js:2831 [Extension → Window][len:     9] Array(9)
/C:/Program Files/Microsoft VS Code/resources/app/out/vs/workbench/workbench.main.js:2831 [Window → Extension][len:     9] Array(9)
/C:/Program Files/Microsoft VS Code/resources/app/out/vs/workbench/workbench.main.js:2831 [Extension → Window][len:    64] Array(64)
/C:/Program Files/Microsoft VS Code/resources/app/out/vs/workbench/workbench.main.js:2831 [Window → Extension][len:    62] Array(62)
/C:/Program Files/Microsoft VS Code/resources/app/out/vs/workbench/workbench.main.js:2831 [Window → Extension][len:     1] Array(1)
/C:/Program Files/Microsoft VS Code/resources/app/out/vs/workbench/workbench.main.js:2831 [Extension → Window][len:     7] Array(7)
/C:/Program Files/Microsoft VS Code/resources/app/out/vs/workbench/workbench.main.js:2831 [Window → Extension][len:     7] Array(7)
/C:/Program Files/Microsoft VS Code/resources/app/out/vs/workbench/workbench.main.js:2831 [Extension → Window][len:     8] Array(8)
/C:/Program Files/Microsoft VS Code/resources/app/out/vs/workbench/workbench.main.js:2831 [Window → Extension][len:     7] Array(7)
/C:/Program Files/Microsoft VS Code/resources/app/out/vs/workbench/workbench.main.js:2830 Extension host terminated unexpectedly. Code:<PHONE_NUMBER>  Signal:  null
O._onExtensionHostCrashed @ /C:/Program Files/Microsoft VS Code/resources/app/out/vs/workbench/workbench.main.js:2830
/C:/Program Files/Microsoft VS Code/resources/app/out/vs/workbench/workbench.main.js:2207 Extension host terminated unexpectedly.
On the insider's version, the error appears after 6-7 seconds, while in the stable build it's instantaneous.
Here's what it says on the insider's console:
Extension host terminated unexpectedly. Code:<PHONE_NUMBER>  Signal:  null - extensionService.ts:377
Extension host terminated unexpectedly. - notificationsAlerts.ts:39
Doesn't work on the insiders also with code-insiders --disable-extensions.
 Code:<PHONE_NUMBER>  Signal:  null
w._onExtensionHostCrashed @ /C:/Program Files/Microsoft VS Code Insiders/resources/app/out/vs/workbench/workbench.main.js:2856
/C:/Program Files/Microsoft VS Code Insiders/resources/app/out/vs/workbench/workbench.main.js:2217 Extension host terminated unexpectedly.
I think you can see that I've done almost everything to get this working. Now please respond and give me a solution to this problem.

Edit
Error does not occur when I remove the "php" folder from the "extensions" folder.

Can you try downloading the Insiders' build and tell me whether it happens there? https://code.visualstudio.com/insiders

Yes it does happen there too, but with a delay of 10-20 seconds. Here's the error there:
/C:/Program Files/Microsoft VS Code Insiders/resources/app/out/vs/workbench/workbench.main.js:2856 Extension host terminated unexpectedly. Code:<PHONE_NUMBER>  Signal:  null
w._onExtensionHostCrashed | @ | /C:/Program Files/Microsoft VS Code Insiders/resources/app/out/vs/workbench/workbench.main.js:2856
/C:/Program Files/Microsoft VS Code Insiders/resources/app/out/vs/workbench/workbench.main.js:2217 Extension host terminated unexpectedly.
e.onDidNotificationChange | @ | /C:/Program Files/Microsoft VS Code Insiders/resources/app/out/vs/workbench/workbench.main.js:2217

Can you also try setting "php.validate.enable": false?

And do you have any other php-related settings set, like the executable path?

No, I do not have any php related settings set. And it does not work even with "php.validate.enable": false. Please help, I am already running behind schedule.

Looking at the code you pointed to it is hard to believe that this can make the extension host crash. And it seems to happen even if he sets "php.validate.enable": false which does spawn PHP at all.

Exactly, it should not be doing much with all the php features disabled.

Is this still an issue? No other reports that I know of.


Add support for IBM Cloud test environment

fixes: #125
This fix contains only the get subcommand, image import and upload may not work. I Will have to work on the s3 api compatibility later as a separate PR.

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: mkumatag
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [mkumatag]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/cc @bpradipt @bkhadars

@bkhadars I have updated the https://github.com/ppc64le-cloud/pvsadm/pull/126#issue-582966682 with more detail with the scope, can you ptal into the PR?

/lgtm


Missing active-directory .md files after master > main migration?

I was just about to submit a PR for a service plan name bug in the below file, but it appears these files no longer exist in the repo?
MicrosoftDocs/azure-docs/master/articles/active-directory/enterprise-users/licensing-service-plan-reference.md
Are they being managed by a different team somewhere else?  The associated public page hasn't been updated in a month either...
https://learn.microsoft.com/en-us/entra/identity/users/licensing-service-plan-reference
Any help finding where to update these is appreciated...
cc @Nicholak-MS

Hi @mpool thanks for bringing this to my attention. I am looking into this.
Thanks
Nick

@mpool
Thanks for bringing this to our attention.
I'm going to assign this to the document author so they can take a look at it accordingly.

Hi @mpool. The repo was recently moved hence the ability to submit a PR was taken offline temporarily. The move is now complete so you should be able to submit PR's once again. Thanks.

#please-close

@Nicholak-MS - Ty!


Dumping ground for things missing in the documentation

Feel free to comment on this issue about concepts, patterns and anti-patterns that Cadence documentation is missing.
We are going to use this thread later to improve the documentation.

From Slack:
_- Don't create workers per request. You could create one at start and then use it.

Your workflow and activities can be with the same worker or even different-remote workers. Time responses will be slower in the second approach though. If you want it faster, use local activities.... etc ... etc..._

Tips for possible configurations: developer (the default), production (arguable subjective but at least some tips for proper configuration)...

Not really "missing", but I would add visually improvement of the documentation. For example increase line height for better readability.

I really missed explanation of the basic client primitives (Future, Select, ...) in a tutorial.


Taking FTC updates for android studio

pulling in changes from parent repo.

See the instructions in discord for updating Android Studio to ladybug.


[RelatedArtists] Use layoutEvent instead of Dimensions

Fixes https://github.com/artsy/eigen/issues/2024
I thought this was addressed but I realized it's one of those layout bugs that can only really be seen on-device. I think this is a pattern we should adopt everywhere because it really doesn't seem like Dimensions is a reliable source for layout.

Nice one!

It seems something is up with CI (it's timing out) so I'm just gonna go ahead and merge.


ES_HEAP_SIZE requirements on different nodes types

It seems excessive that each client, master and data node requires a ES_HEAP_SIZE of 1GB or more of memory.
Is it only the client and master nodes that require so much memory? or just the clients?

It is wrong to assume that just because I have set that value as an example default. Feel free to adapt according to your needs.

Sorry i think maybe my question has been misunderstood.
Im asking if the different node types all have the same memory requirements. e.g.
data, client and master nodes?

Further though leads me to think only the data node really needs extensive memory as its the only node that performs storing and indexing of the documents?
Would that be correct?

It will depend on your use-case and the load you'll expect to handle to the client-nodes. See https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#coordinating-node.

Is it possible to get some best-practice-advice on this issue, we are now running version 5 with:

3 master nodes
2 client nodes
5 data nodes

ES_JAVA_OPTS is set to -Xms1g -Xmx1g, across our cluster that sums to 10GB of RAM allocated. In my opinion this seems excessive and could be reduced, the document referenced above does not really detail situations where work is distributed among nodes with different roles: data, master and slave.
Just seeking some general advice, and perhaps this could be useful to the community in general. :-)

To further elaborate, ES docs state that:

It is important for the stability of the cluster that master-eligible nodes do as little work as possible.

Therefore would it be possible to reduce the heap size for master nodes as they do not require the same resources? Even better for general Kubernetes cluster health as the resources could be allocated to other pods.
Data nodes in the ES docs are described as:

handle data related operations ... These operations are I/O-, memory-, and CPU-intensive. It is important to monitor these resources and to add more data nodes if they are overloaded.

Therefore these nodes require the most resources.

https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html

If there is one resource that you will run out of first, it will likely be memory. Sorting and aggregations can both be memory hungry, so enough heap space to accommodate these is important. Even when the heap is comparatively small, extra memory can be given to the OS filesystem cache. Because many data structures used by Lucene are disk-based formats, Elasticsearch leverages the OS cache to great effect.
A machine with 64 GB of RAM is the ideal sweet spot, but 32 GB and 16 GB machines are also common. Less than 8 GB tends to be counterproductive (you end up needing many, many small machines), and greater than 64 GB has problems

Also, with a machine with x GB, set Elasticsearch heap size to half of it, giving Lucene (the indexing engine) and the OS enough RAM to operate.
See https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html.

The default installation of Elasticsearch is configured with a 1 GB heap. For just about every deployment, this number is usually too small. If you are using the default heap values, your cluster is probably configured incorrectly.

The standard recommendation is to give 50% of the available memory to Elasticsearch heap, while leaving the other 50% free. It won’t go unused; Lucene will happily gobble up whatever is left over.

If you are not aggregating on analyzed string fields (e.g. you won’t be needing fielddata) you can consider lowering the heap even more. The smaller you can make the heap, the better performance you can expect from both Elasticsearch (faster GCs) and Lucene (more memory for caching).

Make sure you read the links I shared.


pull request for e2e test at 01/26/2017 05:45:08

:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Disable pagination

The recent datatables upgrade apparently enabled pagination by default.
We don't query an servers or something and I think our browsers can handle this "big data", so I would suggest to disable paging.
Potential compromise: set the default page length to 500 or 1000 (although then the fact that there is pagination at all would not be so visible)
Appendix
Screenshots
Before

After
😢
The position of the search field is also a bit unintuitive

:wave:
Hi @klyonrad - wasn't pagination always activated it was just further down to the right where it was displayed?
umping the pagination limit might be a good idea anyhow. It'd also magically get rid off that "pagination doesn't work anymore" bug that I hadn't yet thought of :angel:

I just tried it out and at least for 100 files pagination didn't doesn't trigger on the old version... strange I could have sworn... :thinking:

Fixed through https://github.com/colszowka/simplecov-html/pull/96


[CARBONDATA-1109] Cover the case when last page is not be consumed at the end

Problem

We use Producer-Consumer model in the write step, have n(default value is 2 and can be configured) producers and one consumer. The task of generate last page(less than 32000) is added to thread pool at the end, but can't be guaranteed to be finished and put to BlockletDataHolder at the end. Because we have n tasks running concurrently.
We can't guarantee consumer get the last page from BlockletDataHolder at the end.
We have 2 ways to invoke writeDataToFile, one is the size of DataWriterHolder reach the size of blocklet and two is the page is the last page.

So if the last page is not be consumed at the end, we lost the page which be consumed after last page.
Solution
This PR add a flag named isLastPageWrited to make sure every page is writed. If we found after last page is writed, already have other pages, we write the other pages one by one.
cc @jackylk @ravipesala @kumarvishal09

@watermen Can u please add a testcase

test it please

retest this please

retest this please

@kumarvishal09 It is very hard to write the testcase because this case is happened occasionally.

We need make sure all pages of a blocklet are in the right order to be written to carbon data file.

@watermen
This fix is contradictory to the design of producer and consumer. The producer process the data and puts in holder in sequence and consumer takes the data in the same sequence. So for the last set of data there is already a boolean flag writeAll inside NodeHolder enabled to write all the remaining data.
So this fix should not solve the issue, if it solves then there is an issue in producer/consumer design. Please provide the reproduce steps of this issue in the jira.

@ravipesala You can reproduce this case by add some log and run loading sample.csv testcase(TestDataLoadWithFileName).
It's hard to reproducer, so I add sleep in Producer to simulate the time of processing data.

The log lost page like below:
00:15:42 [Thread-65]###addDataToStore
00:15:42 [Thread-65]###addDataToStore
00:15:43 [Thread-73]###Put ---> isWriteAll:false	index:1
00:15:44 [Thread-72]###Put ---> isWriteAll:false	index:0
00:15:44 [Thread-71]###Get ---> isWriteAll:false	index:0
00:15:44 [Thread-71]###Get ---> isWriteAll:false	index:1
00:15:44 [Thread-65]###addDataToStore
00:15:44 [Thread-65]###addDataToStore
00:15:44 [Thread-65]###finish
00:15:46 [Thread-72]###Put ---> isWriteAll:false	index:1
00:15:46 [Thread-72]###Put ---> isWriteAll:true		index:0
00:15:46 [Thread-71]###Get ---> isWriteAll:true		index:0// Last page is not be consumed at the end.
00:15:46 [Thread-71]###Get ---> isWriteAll:false	index:1
00:15:47 [Thread-73]###Put ---> isWriteAll:false	index:0
00:15:47 [Thread-71]###Get ---> isWriteAll:false	index:0

@watermen Thank you for giving reproduce steps, I can reproduce it.
But the fix you have given is not right, This issue happens because of missing semaphore.acquire() in finish method.
So solution is just adding 'semaphore.acquire()' in starting of finish() method solves the issue. Please update the PR.

@ravipesala Thanks for your solution, PR updated.

Build Failed  with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder/2140/

Not required any check, entry count check handled inside
On Fri, 2 Jun 2017 at 11:13 AM, manishgupta88<EMAIL_ADDRESS>wrote:

@manishgupta88 commented on this pull request.
In
processing/src/main/java/org/apache/carbondata/processing/store/CarbonFactDataHandlerColumnar.java
https://github.com/apache/carbondata/pull/978#discussion_r119785140:

@@ -488,13 +488,19 @@ private NodeHolder processDataRows(List dataRows)
public void finish() throws CarbonDataWriterException {
// still some data is present in stores if entryCount is more
// than 0

producerExecutorServiceTaskList.add(producerExecutorService

   .submit(new Producer(blockletDataHolder, dataRows, ++writerTaskSequenceCounter, true)));

blockletProcessingCount.incrementAndGet();
processedDataCount += entryCount;
closeWriterExecutionService(producerExecutorService);
processWriteTaskSubmitList(producerExecutorServiceTaskList);
processingComplete = true;

try {

 semaphore.acquire();

@watermen https://github.com/watermen .....here a check is required for
entryCount > 0.....because we need to acquire a semaphore lock only if
total number of rows in raw data are not exactly divisible by page
size...in this case only we will have some extra rows to be process by
finish method else addDataToStore method will handle all the rows....Please
refer the below code snippet....
public void finish() throws CarbonDataWriterException {
// still some data is present in stores if entryCount is more
// than 0
if (this.entryCount > 0) {
try {
semaphore.acquire();
producerExecutorServiceTaskList.add(producerExecutorService
.submit(new Producer(blockletDataHolder, dataRows,
++writerTaskSequenceCounter, true)));
blockletProcessingCount.incrementAndGet();
processedDataCount += entryCount;
} catch (InterruptedException e) {
LOGGER.error(e, e.getMessage());
throw new CarbonDataWriterException(e.getMessage(), e);
}
}
closeWriterExecutionService(producerExecutorService);
processWriteTaskSubmitList(producerExecutorServiceTaskList);
processingComplete = true;
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/carbondata/pull/978#pullrequestreview-41695875,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHwdxhu_fSOnCMiMXsBYK45_C5Se6qJaks5r_6EWgaJpZM4NrZnd
.

retest this please

Build Failed  with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder/2146/

@ravipesala ....Please correct me if I am wrong....do we need to acquire a semaphore lock if we do not have record to be processed....I think if the rows are exactly divisible we will not have any extra rows to be processed and hence we do not require to acquire semaphore lock and create a new Producer object

retest this please

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder/2147/

@manishgupta88 It was the case in V1 and V2 formats, but in V3 format we should launch producer at finish to flush the older pages it was holding.

LGTM


feat: add  prefix

Description

add prefix to capture mp4

tests are done by build and run rviz2

Pre-review checklist for the PR author
The PR author must check the checkboxes below when creating the PR.

[x] I've confirmed the contribution guidelines.
[x] The PR follows the pull request guidelines.

In-review checklist for the PR reviewers
The PR reviewers must check the checkboxes below before approval.

[ ] The PR follows the pull request guidelines.

Post-review checklist for the PR author
The PR author must check the checkboxes below before merging.

[ ] There are no open discussions or they are tracked via tickets.

After all checkboxes are checked, anyone who has write access can merge the PR.

Please check DCO, and it's better to change the commit message, like feat(tier4_screen_capture_rviz_plugin): add prefix to  video name


fix: Apply image format when saving file in ImagickDriver

After upgrading to the latest version of spatie/image we noticed some issues when converting and cropping PDF files with spatie/media-library. We specify that for each PDF a square JPG thumbnail must be generated, but these turned out to be corrupt.
(I initially believed this was a bug in the spatie/media-library package but I was wrong. The destination files simply weren't JPGs, but PDFs instead. This was weird, as we definitely specify the output format.)
Upon further investigation, it seems like the GD and the Imagick driver handle setting the desired output format slightly differently.
What I discovered was that certain functions like resizeCanvas and overlay in ImagickDriver actually re-assign the image property, thus "losing" the format if it was set prior to calling those transformations (using ->format('jpg'), for example).
You can see the reassignment here. Initially, prior to reassigning $image the format may have been set, and it can be lost afterwards.
A few observations:

I verified that for our use case $image->getFormat() goes from JPG to   (an empty string) after reassignment, which means that the original object's extension will be (incorrectly/unexpectedly) used when actually persisting the image instance.
This is also why this is an issue in combination with spatie/media-library, since there a temp copy of the original file is created, which is then transformed and saved (as the same format because the extension is now used as a fallback).
If the input and output format are the same, you were unlikely to hit this edge case.

This problem can be avoided by ensuring setFormat() on the Imagick image(s) is only called prior to saving the image, at which point the format will always be correct when saving the file.
To do this, the format is stored as a property on the driver instance and is reset when the image is saved. (The GD driver already does it the same way, so I just made a few adjustments to bring this driver in line with the GD driver.)
In order to ensure all tests passed locally, I also had to make sure to check that 'JPEG' is set as the format if the extension or format is specified as 'JFIF', so I added that to the save method as well.
Let me know if I need to make any further adjustments :)
Cheers!

Great work, thanks for the deep investigation and the fix!


Assign FAILED status to a map request after a reasonable timeframe of remaining in PENDING

The MapPacker process sends a message to a mapgl-tile-renderer task worker to create a rendered offline map and updates the database post-rendering, marking it as SUCCEEDED or FAILED. However, issues such as memory or storage limits can kill a renderer container in the middle of a rendering process, leaving the map request status as PENDING indefinitely. To address this, we should detect such failures, set the status to FAILED, provide an error message indicating an unknown error, and write this update directly to the database.
Proposed Implementation:
I'm not sure if we want to set up a dedicated watchdog service to handle this, that seems like overkill; and I think it's better for this concern to sit with map-packer than mapgl-tile-renderer. We can have map-packer perform this check server-side every time  the MapDashboard component is requested by a user, before the page is hydrated.
What should a reasonable timeframe be? I reckon 24 hours is good.

A similar need to monitor the timestamp of a map request is described in https://github.com/ConservationMetrics/map-packer/issues/11.
I might be overestimating how heavy of a lift it is to set up a watchdog service to monitor map request timestamps in the db to set them to status: FAILED or expire them; however, in the latter case, we will need a way to delete files from storage as well.

A Container App Job can be created that runs on a schedule.  It can use the same environment (so same container registery, even same Docker image, same volume mounts) as the mapgl-tile-worker.


E3 sell   flagautosell features

flagautosell features added to the E3 Sell include file.
All of the new features are additions and I avoided modifying any existing routines as much as possible.  Updates to existing routines are in:
SUB sell_Setup
Sub sell_Background_Events
Sub sell_Aliases
These were modified to handle new events, create new INI keys and create new /command aliases.  Everything else should be new features that will not break any existing functionality.
Below is an overview of the new features added and how to use them.
|------------------------------------------------------------------------|
|- flagautosell functionality implemented by CerveloFellow              -|
|- flagautosell will update your Loot Settings.ini file automatically   -|
|-  to Keep,Sell anytime you sell an item to the vendor and the         -|
|-  feature is toggled on                                               -|
|- Several helper functions are available for individual items on your  -|
|- 	/printitemstatus - prints the Loot Settings value for the item on   -|
|-        the cursor.                                                   -|
|- 	/sellthisitem - updates the Loot Settings value for the item on     -|
|-        the cursor to Keep,Sell so that when you invoke /autosell in  -|
|-        future, the item will get sold to the vendor.                 -|
|- 	/keepthisitem - updates the Loot Setings value for the item on      -|
|-        the cursor to Keep so that when /autosell is invoked in       -|
|-        the future, the item will NOT be sold                         -|
|- 	/destroythisitem - updates the Loot Settings value for the item on  -|
|-        the cursor to Destroy so that these items will be destroyed   -|
|-        when /autosell is invoked in the future.                      -|
|-  /flagautosell <true/false> - this will toggle the autosell feature  -|
|-         on or off if you pass an argument.  If no argument is passed -|
|-         this will toggle the current state.  This is mainly for use  -|
|-         when the INI fil Flag Manually Sold Items is Off             -|
|- INI Settings                                                         -|
|-  Flag Manually Sold Items(On/Off)                                    -|
|-    If this setting is On, anytime you sell an item to the vendor, it -|
|-    will get flagged as Keep,Sell in the Loot Settings.ini file. I    -|
|-    reccomend toggling this Off and enabling this feature only when   -|
|-    you want to flag items for autosell.                              -|
|-  Flag Timer(seconds)                                                 -|
|-    When 'Flag Manually Sold Items(On/Off)' is set to off and you     -|
|-    toggle on flagautosell with /flagautosell <True/False>, a timer   -|
|-    starts that will automatically disable this setting afer n        -|
|-    seconds so that you don't acidentally flag stuff for autosell in  -|
|-    the future when selling to a vendor                               -|
|-  Warn on Sell(On/Off)                                                -|
|-     If this setting is On, we attempt to put up a warning anytime you-|
|-     open a merchant window with a vendor and flagautosell is on.     -|
|-    *** This was only tested with a select few merchants in POK and   -|
|-    *** some merchants will not speak when opening a merchant window  -|
|-    *** so the alert popup does not always come up.                   -|
|-     Toggle this to Off if you do not want the warning message to     -|

-     appear                                                           -

My hastily put together PR/commit notes have a few typos.
Some quality of life features added here that I'm call 'flagautosell' features.  These features essentially let you toggle into a state where whenever you sell an item to the vendor, that item will get updated in the Loot Settings.ini file with Keep,Sell so that the next time you /autosell at a vendor, that item will get sold automatically.  This eliminates the need to manually edit the Loot Settings.ini file.
Three new INI entries were added with the following default values:
This first setting is the global flag. When this is On, any items you sell to the vendor will get flagged in Loot Settings as Keep,Sell.  When this is Off, items you sell to the vendor will not be flagged unless you flip a toggle to enter the 'flagautosell' state.
Flag Manually Sold Items(On/Off)=Off
The next INI is for a timer.  When the above INI is set to Off and you toggle into a state to flag items sold to the vendor, a timer is set that will automatically disable the flagautosell state after the timer expires.  This is to help prevent people form inadvertently flagging items to Keep,Sell.
Flag Timer(seconds)=180
This last key is to try and display a warning message when you open a vendor merchant window and you're in the flagautosell state where items sold will get flagged as Keep,Sell.  This will display a message on the screen warning the user that flagautosell is enabled and all items sold will get flagged.   There are some issues with this as not all Merchants say something when you open the merchant window, and events tend to be flaky period with some of the messages.  Not sure how useful this is if it only works some of the time.
Warn on Sell(On/Off)=On
Several helper slash commands have been added
/printitemstatus - when you have an item on your cursor and you issue this command, it will show you the current Loot Settings.ini value for the key.
/sellthisitem - when you have an item on your cursor and you issue this command, it will flag the item in Loot Settings.ini for Keep,Sell
/keepthisitem - when you have an item on your cursor and you issue this command, it will flag the item in Loot Settings.ini for Keep
/destroythisitem - when you have an item on your cursor and you issue this command, it will flag the item in Loot Settings.ini for Destroy
/flagautosell <true/false> - when the default setting is to not flag items(false), you can call this slash command with no parms to toggle the state. Additionaly you can pass as true/false argument with the slash command(true means you'll flag items sold, false means you won't flag items you sell).  This also resets the timer that sets you back to your default state.

Finding quite a big issue while testing this in e3.  The soldItem event fails to trigger fairly often(probably 50% of the time or more).  In a standalone macro, the event triggers 100% of the time.  If anyone has tips on how to get events to get trigger more reliably, I would appreciate the feedback.


UPD added new version of mocha for ragent and console

mocha-junit-reporter version dependency issue was fixed, so we are able to use latest version of mocha

I tried it locally with the same result - build is stuck on
...
✓ uses CA_PATH as path for ca
✓ uses CERT_PATH as path for cert
✓ uses KEY_PATH as path for key
✓ retrieves client options
✓ retrieves server options
58 passing (42s)
issue opened for this: https://github.com/EnMasseProject/enmasse/issues/305

closed without merge because of #305


duperemove: 0.10 -> 0.11

Semi-automatic update generated by https://github.com/ryantm/nixpkgs-update tools.
This update was made based on information from https://repology.org/metapackage/duperemove/versions.
These checks were done:

built on NixOS
Warning: no invocation of /nix/store/mcgis270lv4waah1f68insh0n8j7ja52-duperemove-0.11/bin/btrfs-extent-same had a zero exit code or showed the expected version
/nix/store/mcgis270lv4waah1f68insh0n8j7ja52-duperemove-0.11/bin/duperemove passed the binary check.
/nix/store/mcgis270lv4waah1f68insh0n8j7ja52-duperemove-0.11/bin/hashstats passed the binary check.
/nix/store/mcgis270lv4waah1f68insh0n8j7ja52-duperemove-0.11/bin/show-shared-extents passed the binary check.
2 of 4 passed binary check by having a zero exit code.
2 of 4 passed binary check by having the new version present in output.
found 0.11 with grep in /nix/store/mcgis270lv4waah1f68insh0n8j7ja52-duperemove-0.11
directory tree listing: https://gist.github.com/0df1e551a8c77ca4dd4bb8f2783f75c4
du listing: https://gist.github.com/4dc88fc1b6adba1caf3a67e67383c3c9

meta.description for duperemove is: '"A simple tool for finding duplicated extents and submitting them for deduplication"'.
cc @bluescreen303 @thoughtpolice for testing.

No attempt on x86_64-darwin (full log)
The following builds were skipped because they don't evaluate on x86_64-darwin: duperemove
Partial log (click to expand)

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

Success on aarch64-linux (full log)
Attempted: duperemove
Partial log (click to expand)
shrinking /nix/store/qddpl7js6mv84d0i254x7i1ha998iqh2-duperemove-0.11/sbin/btrfs-extent-same
shrinking /nix/store/qddpl7js6mv84d0i254x7i1ha998iqh2-duperemove-0.11/sbin/hashstats
shrinking /nix/store/qddpl7js6mv84d0i254x7i1ha998iqh2-duperemove-0.11/sbin/duperemove
gzipping man pages under /nix/store/qddpl7js6mv84d0i254x7i1ha998iqh2-duperemove-0.11/share/man/
strip is /nix/store/jk6j4lh9v5mvjdbdc35sj0zffhhf6s56-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/qddpl7js6mv84d0i254x7i1ha998iqh2-duperemove-0.11/sbin
patching script interpreter paths in /nix/store/qddpl7js6mv84d0i254x7i1ha998iqh2-duperemove-0.11
checking for references to /build in /nix/store/qddpl7js6mv84d0i254x7i1ha998iqh2-duperemove-0.11...
moving /nix/store/qddpl7js6mv84d0i254x7i1ha998iqh2-duperemove-0.11/sbin/* to /nix/store/qddpl7js6mv84d0i254x7i1ha998iqh2-duperemove-0.11/bin
/nix/store/qddpl7js6mv84d0i254x7i1ha998iqh2-duperemove-0.11


Make ghost warp use AttachToGridOrMap()

Ensures that ghost warping uses AttachToGridOrMap() to avoid getting stuck being parented to other entities. Also adds an option to warp points that will cause observers to follow the entity, rather than just warp to it.

this should probably just change their map position rather than entity coordinates


"xenserver has no compute service"

When using fog-xenserver with Foreman as a compute resource, I get this message when trying to list vms:
There was an error listing VMs: xenserver has no compute service

Is this plugin actively maintained anymore? What can I do to spark life back into it?

Yes. It is being maintained.
Can you provide more feedback on the issue?

It does not have heavy development anymore because it does support almost the whole API already if not all. Whats left are bugs i believe.

Can you post the version og fog adn fog-core?

and we run this gem in production 😎

@plribeiro3000
[root@c4katello01 gems]# ls | grep fog
fog-1.37.0
fog-aws-0.7.4
fog-azure-0.0.2
fog-core-1.35.0
fog-google-0.1.0
fog-json-1.0.2
fog-vsphere-0.6.3
fog-xenserver-0.2.3
fog-xml-0.1.2

Can you try the latest fog-core (1.42.0)?

I believe I just updated to the latest fog-core, and the issue persists.
Here's what I believe are relevant log lines.
2016-08-16 16:28:26 [app] [W] Error has occurred while listing VMs on C4 Xen (Xenserver)
 | Fog::Service::NotFound: xenserver has no compute service
 | /opt/rh/rh-ruby22/root/usr/local/share/gems/gems/fog-core-1.42.0/lib/fog/core/services_mixin.rb:18:in `rescue in new'
 | /opt/rh/rh-ruby22/root/usr/local/share/gems/gems/fog-core-1.42.0/lib/fog/core/services_mixin.rb:8:in `new'
 | /opt/rh/rh-ruby22/root/usr/local/share/gems/gems/fog-core-1.42.0/lib/fog/compute.rb:54:in `new'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_xen-0.3.0/app/models/foreman_xen/xenserver.rb:367:in `client'
 | /usr/share/foreman/app/models/compute_resource.rb:136:in `vms'
 | /usr/share/foreman/app/controllers/compute_resources_vms_controller.rb:4:in `index'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/abstract_controller/base.rb:189:in `process_action'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_controller/metal/rendering.rb:10:in `process_action'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:113:in `call'

I checked the connection initialization here and it seems to be doing it right.
The fact that foreman_xen has fog-xenserver as a dependency here it should have been loaded correctly by bundler.
Since fog-xenserver work as standalone and as a dependency of fog i'm out of ideas why it does not work inside foreman.  Is there any other piece of code you can share that might be related? Is it breaking only on this feature or anything else works?

I'm gonna keep trying to dig into this myself and see if I can get it to work. Gonna spin up an instance on my local machine to see if I can reproduce the issue.
It is, for the record, all features that cause this error to be shown. :(

So I've just installed and setup a new foreman server and installed this plugin... but I still get "xenserver has no compute service". The only thing I changed is editing the connection.rb file in lib/fog/xen_server to change the XMLRPC call to use SSL over port 443--since we're sending passwords here and it's probably not safe to just send them over HTTP... plus I've only opened :443 through the firewall.

Any thoughts on where I could begin to look at this? I see that it's not just the xenserver fog plugin.

@NeilHanlon My guess it is related to fog-core and service loading.
Did you try another version of the lib?

@plribeiro3000 I've upgraded to the most recent versions, but that doesn't seem to change anything, unfortunately. Should I try downgrading versions? Any specific one you think I should target?

None i can think of from the top of my head. =s

I've figured this out. Long story short, I tried to modify the code to connect over SSL, but didn't understand how function arguments work in ruby.. so I tried just adding a use_ssl=true to the XMLRPC call for .new(), but this was setting port=true. Down the line, it tried to run @port.to_i, which created a NoMethodError. This was finally caught by the foreman_xen gem as a NameError, and rethrown.
I'll probably be putting a PR against this to add SSL support.

So this is not a fog-xenserver bug, right?
Can i close this ticket?

It is not a bug directly. It was caused by the need for a feature.
In short, yes. I will close this.

Hey Guys, This is still not sorted out. I am using XenServer 7.2 with Foreman 1.19.
Was a PR ever created to fix this? Is the short term fix to turn off ssl?

@NeilHanlon issue was related to something else completely if i'm not mistaken.
And no, no one has worked on this yet. You are welcome to send a PR if you find some spare time!


Tzfile support

So that we don't have to separately ship something like pytz rely on the fragile, sometimes incorrect and unreliable OS APIs :)

I may be interested in tackling this with some guidance @lifthrasiir, mainly since I need to have DST support in Chrono. Would that be ok?
And do you still have your branch with basic tzfile handling somewhere?

@ehsanul Yes, I still have a branch around; the main blocker was several edge cases around TzRule (which is a part of TzFile v3), and Chrono 0.3 (still slowly in development, sorry :S) would have necessary changes to avoid such edge cases.
For now, though, you can make an individual type for time zones observing DST.

Tzif support including TzRule was added in https://github.com/chronotope/chrono/pull/677.
I think it would be could if this could be extended to more timezones than just Local, but that is not what this issue is about.