ISPN-11896 non blocking cache store SPI docs

https://issues.redhat.com/browse/ISPN-11896

@wburns hi. these are all just suggestions for phrasing along with stuff like grammar + punctuation. feel free to leave or take anything. I'd also be happy to go through anything here and explain some of the suggested changes.
there are places where I've changed passive voice to active voice. also changed tense from future to present, which can help improve clarity. but I've been careful to avoid changing meaning, so hopefully there's nothing here that loses technical accuracy.

Integrated into master, thanks @oraNod !


Code generation for a project written in French: Bad client name

I am working on a project written in French.
The classes are named in French.
I want to use code generation for a C# client.
[Route("[controller]s")]
public class EtablissementControleur : ControllerBase
{
  ...
}

My api route is good and my swagger documentation is good, because I took care of the translation.
/api/v1/etablissements/{id}/adresse
Here is the partial configuration of the generator
nswag.json
  "codeGenerators": {
    "openApiToCSharpClient": {
      "className": "{controller}HttpService",
...

The name of the generated service is EtablissementControleurHttpService, whereas I was expecting EtablissementHttpService.
    [System.CodeDom.Compiler.GeneratedCode("NSwag", "<IP_ADDRESS> (NJsonSchema v<IP_ADDRESS> (Newtonsoft.Json v<IP_ADDRESS>))")]
    public partial class EtablissementControleurHttpService : IEtablissementControleurHttpService
    {

Should I tell the generator to remove the word Controller?
Or should I modify the swagger documentation to support it?
Thanks for your help

it removes Controller not Controleur


[build-script] XCTest xcodebuild uses SWIFT_EXEC

What's in this pull request?

Because the xcodebuild invocation used to compile swift-corelibs-xctest on OS X uses a toolchain that is usually out of date, it often fails (for example, when building the newly migrated Swift 3 code).
Pass the path to the freshly built swiftc to xcodebuild in order to use the latest Swift compiler. This should allow the swift-corelibs-xctest OS X CI to pass.
Resolved bug number: None

Before merging this pull request to apple/swift repository:

[ ] Test pull request on Swift continuous integration.

Triggering Swift CI
The swift-ci is triggered by writing a comment on this PR addressed to the GitHub user @swift-ci. Different tests will run depending on the specific comment that you use. The currently available comments are:
Smoke Testing
    Platform     | Comment
    ------------ | -------------
    All supported platforms     | @swift-ci Please smoke test
    OS X platform               | @swift-ci Please smoke test OS X platform
    Linux platform              | @swift-ci Please smoke test Linux platform

Validation Testing
    Platform     | Comment
    ------------ | -------------
    All supported platforms     | @swift-ci Please test
    OS X platform               | @swift-ci Please test OS X platform
    Linux platform              | @swift-ci Please test Linux platform

Note: Only members of the Apple organization can trigger swift-ci.

@shahmishal Would it be possible to trigger the "buildbot_incremental,tools=RA,stdlib=RA,XCTest" preset on OS X and Linux with these changes? I'd like to see if it gets tests passing.

@swift-ci Please test

The Linux SIL/Parser/undef.sil test failure seems unrelated. I imagine CI is going to be pretty rocky with Swift 3 having just been merged; perhaps we could revisit this at a calmer time.

@modocache Yes, lets test buildbot_incremental,tools=RA,stdlib=RA,XCTest preset once master is stable again.

@swift-ci Please test


Query requests doesn't work on nunit in netStandard project

Hello. I have NetStandard Library, where my code.
I have .Net Class Library with tests on Microsoft.VisualStudio.TestTools.UnitTesting. All tests work very well.
But another NetStandard Class Library with tests on NUnit doestn't work in some cases.
Requests by linq not works. Examples:

Objects not deleting.

using (var session = objectsStore.OpenSession())
{
        session.DeleteWhere<TElement>(o => true);
        session.SaveChanges();
}

Query<> requests doesn't works

using (var session = objectsStore.QuerySession())
{
        var workRequest_1 = session.Load<TElement>("119569a16e9b4301b629e162dcbeedf7"); // result - my stored object
        var workRequest_2 = session.Query<string>("Select * FROM public.mt_doc_testobjects").ToList(); // result - all stored objects as string
        var empty_1 = session.Query<TElement>().ToList();
        var empty_2 = session.Query<TElement>().Where(o=>o.Id!=null).ToList();
}

@KopIlya,
session.Query<string>() isn't supported. And as for the rest, I don't know. There's nothing about the testing tools that would impact Marten that way. There's not really enough information here to help you much.

@kopIlya can you please elaborate what were the type detail for TElement in each of the cases which you have described?

@KopIlya can you please elaborate what were the type detail for TElement in each of the cases which you have described?

public class TestObject : IStorableObject
{
}

public interface IStorableObject
{
    string Id { get; set; }
    string SystemId { get; set; }
    int Version { get; set; }
}

There's no recent activity in this issue, I'm closing it.


Opens the Select File dialog box. When closed, garbled!

Opens the Select File dialog box. When canceled, return garbled!
path := w.Dialog(webview.DialogTypeOpen,0,"ni hao ","dialog")
ret.Data=path
log.Println(path)

���n�Epache-maven-3.3.9\NOTICE

webview.h ---> 1392
 if (result != NULL) {
      result[0] = '\0';
  }


Hot-reloading

I'm curious if there's a plan (or way) to hot-reload the site after compiling the SASS like node can.

I think you are confused on what Node.js is. Node is just JavaScript that can be ran outside of a browser and has some extra features (I/O, hardware, OS, networking access).
There are modules (libraries) written with Node that allow for live reloading or hot reloading (not the same thing).
There is a plan to offer live reloading in a future version of Scout-App.
Closing this issue as a duplicate of #39.


Added function to compute the CV2 (coefficient of variation for non-stationary time series)

Added CV2 (coefficient of variation for non-stationary time series Reference: http://jn.physiology.org/content/jn/75/5/1806.full.pdf) to the statistics module

Coverage decreased (-0.03%) to 90.866% when pulling a8a8e6691919adf008684754ad73d957078925e3 on INM-6:feature/cv2 into 93169704d6bde21f4fdc7a27079c90503a7b65a7 on NeuralEnsemble:master.

Coverage increased (+0.02%) to 90.918% when pulling a8a8e6691919adf008684754ad73d957078925e3 on INM-6:feature/cv2 into 93169704d6bde21f4fdc7a27079c90503a7b65a7 on NeuralEnsemble:master.


One Logging Bridge per language - Ruby

For anyone who comes across this issue, please look at the Logs project board for the current status of the project.


RFC: Feedback on API concept

Creating this issue to gather feedback on the API concept introduced in https://github.com/spotify/backstage/pull/251 and https://github.com/spotify/backstage/pull/326.
Anything goes, questions about purpose, ideas, clarifying things, naming, alternatives, etc.
Starting out with providing a few answers:
Why was this created in the first place?
In the Spotify internal Backstage we added the concept as a way to make it easier to write tests, and to wrangle the growing complexity of the dependency graph between different app-wide features.
What is the purpose of the API concept?
Apart from what's mentioned above, it's now also provides a way to decouple plugins from the APIs that they consume. A plugin can for example depend on an error reporting API, and it is up to the app that uses the plugin to provide an implementation of the API.
Would every app have to write their own API implementations?
The backstage/core, or possibly a separate package, would ship a set of default implementations for all APIs. Apps can choose to use these directly, build upon them, or replace them.
How does the API concept help with testing?
The backstage/core package would include mocks for all core API definitions. They should provide sane defaults for all behaviour and stay out of the way, but also provide additional methods for testing error/edge cases.
For example, any component consuming a core API in a way where we don't want to do any further testing of the API would simply do something like
const rendered = render(wrapInTestApp(<MyComponent />))

While for example the error api mock could provide methods for ensuring that errors have been reported, something like
const errorApi = testRegistry.get(errorApiRef) as MockErrorApi;

const rendered = render(wrapInTestApp(<MyComponent />))

expect(errorApi.reportedErrors.length).toBe(1);

Are the API definitions limited only to the ones listed in backstage/core?
For now it would be the only supported way, but we want to support externally defined APIs, and APIs defined and consumed within a single plugin.

I've said this before but if you want to go take your security to the next level (this could fe be done in a v2 rewrite of Backstage), you should make each API and plugin live within it's own javascript namespace. This is typically done by leveraging the same-origin policy of browsers (eg. through iframes).
More practically speaking:
Your dns server forwards all traffic from *.backstage.cronus.com to the Backstage server. The backstage server will serve the core application from core.backstageorigin.com. Plugins would be served from <plugin>.plugin.backstageorigin.com.
Let's say we have an authenticator API, it would be loaded in the browser under an iframe with src="https://auth.plugin.backstageorigin.com". Now all the authentication internals (and the cookies+storage) are completely isolated from the other plugins. The only communication that is permitted is communication through window/postMessage.
Take this a step further and you can require plugins to declare a manifest file with the plugins they want to communicate with (this should be done anyways for dependency management), now the core can enforce that only plugins that need to communicate with eachother can communicate with eachother.
Note: This is how my ideal backstage world would look like. I assume there are not sufficient resources for an implementation like this for now, hence 'v2'.

@toonsevrin Yeah I think the declaration of dependencies in a manifest is something to explore as a part of the API concept. it will be needed in some capacity so that whoever creates a backstage app knows what APIs have to be implemented based on what plugins are being used.
Regarding separated origins I'm not so sure tbh, I think leaning towards read-only interface with destructive actions in a secure context has a better security/UX/DX tradeoff.

I have have a naming-help request related to this API concept! We've tried some different terminology, like "services", "factories", "libraries", but APIs is short and matches the concept, so it's what we've gone with for now.
The request is to find a word to describe these APIs though, as saying simply "Backstage API" is a bit confusing, as in what kind of API are we talking about? So looking for a word to place before/between "Backstage" and "API", to be able to refer to for example the existing error API as a "<blank> API". Or talk about "Backstage <blank> APIs".
Some ideas: "frontend", "plugin", "integration", "provided", "app"
Or a bit more abstract: "fuse", "axial", "edge", "perimeter"

I like 'Backstage Plugin APIs' for plugin concerns and 'Backstage App APIs' for app concerns.
Ex: Would error API be only used by plugins or the app can also use it to push error messages to the user? What I am getting here is can APIs be distinguished by usage or there will be overlaps?

+1 to Backstage Plugin APIs

@soapraj @wesleyyee I think usage of APIs would really extend everywhere, including apps, plugins, and shared components, so yeah there would be overlap. this being an example of that
Thing that worries me about prefixing with "plugin" or "app" is that it would be confused with createPlugin and createApp etc.
Still don't have anything better than "plugin APIs" though :p
"Backstage Global API", "Backstage Universal API"? 😅

I also like Provider or Utility

Yeah "Utility API" isn't bad. It gets a bit weird when you talk about something like the "Lighthouse Utility API", but I don't think we'll find a name that fits perfectly with every use-case.
"Provide(r/d) API" gets tricky because of the clashes with React context imo.

Another option that I think covers a couple more use-cases is "Service APIs". Might help people coming from Angular as well, as they're similar and used in the same way as angular services.

Discussed the naming a bit further today and settled for trying out "Utility APIs". Service APIs is too easy to confuse with the "service" component entity type, and we want to keep these Utility APIs limited to only things that you access using the useApi hook. With that separation in mind, "Plugin APIs" or "App APIs" would be too easy to mix up with createPlugin, createApp and friends.


[Request ] Please add a method to halt the execution of a cell

Steps to Reproduce

Create a cell with long-running code (eg. a number of HttpClient fetches, etc.)
Try to find a way to stop it! ;)

NB. Either the way to stop execution is non-obvious, or (as I suspect) there isn't yet a way to do it. It would be extremely handy! Thanks!
Xamarin Workbooks

Detail
Value

Version
1.4.3

Git Branch
1.4-release

Git Hash
db27525

VSTS Definition
Workbooks & Inspector (Win)

System Info

Component
Value

Windows
Microsoft Windows NT 10.0.17134.0 (64-bit)

CPU Cores
4

Physical Memory
16 GB

System-Installed Software

Component
Version

Xamarin iOS
10.10.0

Xamarin Android
7.3.0-13

You can hover over the "running" indicator, and reveal an "abort" button.

Hope this helps; feel free to re-open if this still doesn't work for you.

This button doesn't seem to be working for me or it's so delayed that it seems like it's not working. I suspect there may be a threading issue or something where the abort is being blocked, perhaps?


Assign Hello-World job to correct docker worker

The docker worker has the label "docker-jnlp" not "docker-worker-1"
Pair: @poveyd & @smford

Merged as done as a pair.


Seeing this engine just makes me depressed

Legit I thought I had the brand new next big thing to do that'll help the FNF modding community a ton by adding 3D getting rid of haxeflixel throwing that shit out the window in favor of a custom game engine written in rust but when I saw the ram usage of this engine I realized that someones already done better things then me, someones already made something better
No matter how hard I try i'll always be beat and without hundreds of hours at my disposal I won't be able to compete with this engine :(
I shouldn't even call myself a programmer compared to people like the CNE devs
I know this isn't an issue but just wasn't to say that congrats on making a really good engine

yes you can believe in you're self and plus i may not know you but you must have a better personality then yoshi crafter tbh 👍🏻

i think that's what someone said when they saw kade engine for the first time.
don't give up. keep working on your engine chum

This is making me sad to read.

the fact that theres another engine that you think is "better" out there shouldnt be whats stopping you from continuing yours
you arent a bad developer, the fact you made an entire game engine in rust is amazing, hell it prob runs better than codename does
i also felt unmotivated from times to times, mainly due to the fact people are still going to stick to psych even if codename comes out, but have i stopped? no.
if its a passion for you, continue. dont let anything else get through your path. you dont need to compete with codename, youre already succeeding in your own way
hope that makes u feel better


How to get notified of websockets that close

When a websocket connection closes I often see:
03.07.21 22:35:58.492      dream.http  WARN WebSocket (<IP_ADDRESS>:33826): Unix.Unix_error(Unix.ECONNRESE)
03.07.21 22:35:58.492      dream.http  WARN Raised by primitive operation at Lwt_unix.shutdown in file 8
03.07.21 22:35:58.492      dream.http  WARN Called from Gluten_lwt_unix.Io.close.(fun) in file "src/ven7
03.07.21 22:35:58.492      dream.http  WARN Called from Lwt.Sequential_composition.catch in file "src/c0

But I cannot react to that. I only know that a connection is not there anymore when I try to send something to it and that fails. I should explore this a bit more, but it seems that Dream.receive does not always get a None when the connection is closed.

But I cannot react to that.

You should be able to catch the rejection of Dream.send with ECONNRESET.

I should explore this a bit more, but it seems that Dream.receive does not always get a None when the connection is closed.

This issue seems to be about what happens when you call Dream.send on a socket that has been closed by the client, rather than what happens when you call Dream.receive on it. Dream.send needs some way to tell you that the socket has been closed  by the other side. ECONNRESET is the standard "systems programming" way of doing that.
I can agree, however, that it's not a very ergonomic interface, and perhaps we can think of something better. However

We do need to tell the sending code that the send failed.
We need then some other "centralized" way to communicate closing of the socket by the client.

We do need to tell the sending code that the send failed.

I guess it's also possible to never resolve the promise that comes from Dream.send, relying on the client handling some kind of other "remote close" notification.

I have this code:
let send client client_id payload =
  try%lwt Dream.send client payload with
  | e ->
      Stdlib.print_endline (Exn.to_string e) ;
      log.error (fun log -> log "failed to send to client: %i" client_id) ;
      Lwt.return_unit

And sometimes my log only prints:
04.07.21 14:00:59.083      dream.http  WARN WebSocket (<IP_ADDRESS>:33880): Unix.Unix_error(Unix.ECONNRESE)
04.07.21 14:00:59.083      dream.http  WARN Raised by primitive operation at Lwt_unix.shutdown in file 8
04.07.21 14:00:59.083      dream.http  WARN Called from Gluten_lwt_unix.Io.close.(fun) in file "src/ven7
04.07.21 14:00:59.083      dream.http  WARN Called from Lwt.Sequential_composition.catch in file "src/c0

and sometimes only:
(Failure "cannot write to closed writer")
04.07.21 14:01:38.647  dream.channels ERROR REQ 2 failed to send to client: 0

So I have the feeling that the try%lwt not always catches the exception, but I guess that would be very weird, so it's probably something in my code?
(the reason for the Stdlib.print_endline was because I didn't always see the log.error but it does work, so I can remove it again)
Anyway why not change Dream.send to return an Lwt_result.t and also offer Dream.send_exn?

For now, I'll wrap Dream.send to return a result.
I have 2 remaining questions though:

Can Dream.send fail for an other reason than that the client is disconnected?
Can dream detect a broken websocket when there is nothing being send/received. So say a clients internet connection drops and nothing is send to that client?

Can Dream.send fail for an other reason than that the client is disconnected?

For example, if you do Dream.send before the previous Dream.send's promise has resolved.

2. Can dream detect a broken websocket when there is nothing being send/received. So say a clients internet connection drops and nothing is send to that client?

I don't believe so. In general, the assumption is that you'd always be either sending or receiving on a WebSocket. It's the same API as for systems sockets, but we probably shouldn't duplicate it verbatim since we don't have to, for the reasons your question is implying :)

For example, if you do Dream.send before the previous Dream.send's promise has resolved.

Good to know! (This would be nice if this was added to the documentation / or a result with some self documenting type)

I don't believe so. In general, the assumption is that you'd always be either sending or receiving on a WebSocket. It's the same API as for systems sockets, but we probably shouldn't duplicate it verbatim since we don't have to, for the reasons your question is implying :)

So it's totally possible that broken connections would linger forever if you never send to them?
That doesn't seem right? Shouldn't we need some mechanism to clean these up?
For example, using a slightly adapted w-chat example but connecting with a node client (https://github.com/websockets/ws/blob/master/doc/ws.md#websocketterminate.)
When I create multiple ws clients and call ws.terminate() node will destroy the socket, but this is not always received by dream.
let send_exn client payload =
  try%lwt Dream.send client payload with
  | error ->
      let error = Printexc.to_string error in
      log.error (fun log -> log "failed to send to client - error: %s" error) ;
      Lwt.return_unit

let send message =
  Hashtbl.to_seq_values clients
  |> List.of_seq
  |> Lwt_list.iter_p (fun client -> send_exn client message)

let handle_client websocket =
  let client_id = connect websocket in
  let rec loop () =
    match%lwt Dream.receive websocket with
    | Some message ->
      let%lwt () = send message in
      loop ()
    | None ->
      log.info (fun log -> log "disconnection %i" client_id) ;
      disconnect client_id;
      Dream.close_websocket websocket

...
// starting ws1 to ws3 and sending 1 message
ws1.terminate();
ws2.terminate();
ws3.terminate();

I get this output:
04.07.21 21:24:06.679                       Running on <IP_ADDRESS>:8080 (http://localhost:8080)
04.07.21 21:24:06.679                       Type Ctrl+C to stop
04.07.21 21:24:09.174       dream.log  INFO REQ 1 GET /ws <IP_ADDRESS>:34640
04.07.21 21:24:09.174       dream.log  INFO REQ 1 101 in 50 μs
04.07.21 21:24:09.174       dream.log  INFO REQ 2 GET /ws <IP_ADDRESS>:34646
04.07.21 21:24:09.174       dream.log  INFO REQ 2 101 in 21 μs
04.07.21 21:24:09.174       dream.log  INFO REQ 3 GET /ws <IP_ADDRESS>:34642
04.07.21 21:24:09.174       dream.log  INFO REQ 3 101 in 18 μs
04.07.21 21:24:09.177      dream.http  WARN WebSocket (<IP_ADDRESS>:34640): Unix.Unix_error(Unix.ECONNRESET, "read", "")
04.07.21 21:24:09.177      dream.http  WARN Raised by primitive operation at Lwt_unix.shutdown in file "src/unix/lwt8
04.07.21 21:24:09.177      dream.http  WARN Called from Gluten_lwt_unix.Io.close.(fun) in file "src/vendor/gluten/lw7
04.07.21 21:24:09.177      dream.http  WARN Called from Lwt.Sequential_composition.catch in file "src/core/lwt.ml", 0
04.07.21 21:24:09.177      dream.http  WARN WebSocket (<IP_ADDRESS>:34646): Unix.Unix_error(Unix.ECONNRESET, "read", "")
04.07.21 21:24:09.177      dream.http  WARN Raised by primitive operation at Lwt_unix.shutdown in file "src/unix/lwt8
04.07.21 21:24:09.177      dream.http  WARN Called from Gluten_lwt_unix.Io.close.(fun) in file "src/vendor/gluten/lw7
04.07.21 21:24:09.177      dream.http  WARN Called from Lwt.Sequential_composition.catch in file "src/core/lwt.ml", 0
04.07.21 21:24:09.177      dream.http  WARN WebSocket (<IP_ADDRESS>:34642): Unix.Unix_error(Unix.ECONNRESET, "read", "")
04.07.21 21:24:09.177      dream.http  WARN Raised by primitive operation at Lwt_unix.shutdown in file "src/unix/lwt8
04.07.21 21:24:09.177      dream.http  WARN Called from Gluten_lwt_unix.Io.close.(fun) in file "src/vendor/gluten/lw7
04.07.21 21:24:09.177      dream.http  WARN Called from Lwt.Sequential_composition.catch in file "src/core/lwt.ml", 0

As you can see, I get no disconnection whatsoever here, sometimes I do get 1 or 2 disconnections, but I have never seen 3.
So here, these broken sockets stay alive until we send to them and detect the problem.

Phoenix has a ping/pong (https://hexdocs.pm/phoenix_gen_socket_client/Phoenix.Channels.GenSocketClient.Transport.WebSocketClient.html) but it requires client support of course.
I'm working (very WIP - not public yet) on dream-channels - Phoenix like channels for dream and one of my goals is to have a client library as well.
Maybe I could add it in there? Any thoughts

I'll have to take a very close look at this as I'm doing the I/O internals rework, which I intend for the next alpha release.

Phoenix has a ping/pong (https://hexdocs.pm/phoenix_gen_socket_client/Phoenix.Channels.GenSocketClient.Transport.WebSocketClient.html) but it requires client support of course.
I'm working (very WIP - not public yet) on dream-channels - Phoenix like channels for dream and one of my goals is to have a client library as well.
Maybe I could add it in there? Any thoughts

WebSockets already have their own low-level ping-pong built into the protocol, and Dream already responds to the client's WebSocket pings, but doesn't send its own pongs:
https://github.com/aantron/dream/blob/7e4dae59fd17922c29c41df3c323b81454a2eed0/src/http/http.ml#L88
I suspect (but don't know, could be way off) Phoenix has higher-level ping-pong (if it does) because it can use non-WebSocket transports for channels (is that still/was ever true?). IIRC Phoenix could use server-sent events and maybe other methods to get channel behavior. Given that, it makes sense that Phoenix would implement a Phoenix-level ping-pong, since SSE doesn't have anything like that built into the "protocol."
Perhaps one thing we need is for Dream to send low-level pings to WebSocket clients, probably with some optionals for configuration.

Here's the Dream server-sent events example in action by the way, but I don't think we should rely on this. I suspect (again, could be way off) that Phoenix supports SSE as a transport for channels because Phoenix channels were written at a time when WebSockets were still very new, and not that well supported yet. We can probably just use only WebSockets at this point.

Phoenix supports SSE as a transport for channels because Phoenix channels were written at a time when WebSockets were still very new, and not that well supported yet

I believe that's true, Phoenix has a fallback for longpolling for example.

We can probably just use only WebSockets at this point.

Agree!

WebSockets already have their own low-level ping-pong built into the protocol, and Dream already responds to the client's WebSocket pings, but doesn't send its own pongs:

awesome, didn't know that. I need to start looking into the websocket protocol more :-)


chore: create separate gradle project for producing java proto artifacts

Currently, we have a single jar artifact in the "kotlin-messages"
gradle project which bundles both java and kotlin proto artifacts.
We are also publishing this to our internal codeartifact repo.
This commit is the first of a series to clean this up and get these
published to maven central so that users can build our OSS projects
and we can get rid of the submodules on the Java SDK.
This commit introduces a new java gradle project that builds a jar
that contains only the java proto artifacts.  Subsequent commits
will include:

Publishing this java protos jar artifact to maven central.
Removing the java protos from the kotlin jar.
Removing the codeartifact stuff from the kotlin jar and publishing
that jar to maven central.
Updating the java SDK to get rid of the submodules and consume
the jar from maven central.
Updating any places in the service code where we are referring
to the kotlin artifact, to use the maven central coordinates
instead.

Left a comment on the other one

@kvcache said:

This looks good. Did you intend to skip the .github pr workflow though? Seems to me like that one wouldn't need any publishing path worked out

No I didn't intend to skip the PR workflow.  Can do it in next PR, or this one if it isn't merged by the time I have bandwidth for it.

@kvcache now has CI for java lib


initial cleanup

Description
Remove unwanted packages in core, modules and tests and make more packages optional, all based on @xmonader suggestions.
Changes

Remove bcdb (also client, tests and example scripts).
Remove data.types, schemac and related tool
Remove pylzma
Remove usershell
Merge mongodb store backend from #522.
Make more packages optional and installable via extras poetry install -E mongo (e.g. whoosh, mongo, dill, docker, git...).

Checklist

[x] Pre-commit hook is installed to do formatting checks before committing code...etc
[x] Tests included
[x] Build pass
[x] Documentation
[x] Code format and docstrings

Codecov Report

Merging #537 (5340db6) into development (2cca5d2) will increase coverage by 0.22%.
The diff coverage is 66.66%.

@@               Coverage Diff               @@
##           development     #537      +/-   ##
===============================================
+ Coverage        70.41%   70.64%   +0.22%
===============================================
  Files               84       82       -2
  Lines             3820     3665     -155
===============================================
- Hits              2690     2589     -101
+ Misses            1130     1076      -54

Impacted Files
Coverage Δ

jumpscale/core/config/config.py
93.33% <ø> (ø)

jumpscale/shell/config.py
0.00% <0.00%> (ø)

jumpscale/data/serializers/__init__.py
80.00% <50.00%> (-20.00%)
:arrow_down:

jumpscale/loader.py
89.09% <50.00%> (-6.75%)
:arrow_down:

jumpscale/sals/nettools/__init__.py
22.94% <50.00%> (+1.73%)
:arrow_up:

jumpscale/core/base/factory.py
91.23% <63.63%> (-1.20%)
:arrow_down:

jumpscale/core/base/store/mongo.py
95.00% <95.00%> (ø)

jumpscale/clients/git/__init__.py
80.00% <100.00%> (+5.00%)
:arrow_up:

jumpscale/data/serializers/pickle.py
100.00% <100.00%> (ø)

... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2cca5d2...5340db6. Read the comment docs.

Codecov Report

Merging #537 (5340db6) into development (2cca5d2) will increase coverage by 0.22%.
The diff coverage is 66.66%.

@@               Coverage Diff               @@
##           development     #537      +/-   ##
===============================================
+ Coverage        70.41%   70.64%   +0.22%
===============================================
  Files               84       82       -2
  Lines             3820     3665     -155
===============================================
- Hits              2690     2589     -101
+ Misses            1130     1076      -54

Impacted Files
Coverage Δ

jumpscale/core/config/config.py
93.33% <ø> (ø)

jumpscale/shell/config.py
0.00% <0.00%> (ø)

jumpscale/data/serializers/__init__.py
80.00% <50.00%> (-20.00%)
:arrow_down:

jumpscale/loader.py
89.09% <50.00%> (-6.75%)
:arrow_down:

jumpscale/sals/nettools/__init__.py
22.94% <50.00%> (+1.73%)
:arrow_up:

jumpscale/core/base/factory.py
91.23% <63.63%> (-1.20%)
:arrow_down:

jumpscale/core/base/store/mongo.py
95.00% <95.00%> (ø)

jumpscale/clients/git/__init__.py
80.00% <100.00%> (+5.00%)
:arrow_up:

jumpscale/data/serializers/pickle.py
100.00% <100.00%> (ø)

... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2cca5d2...5340db6. Read the comment docs.

Should checkout current changes to nettools sal (already covered in #531).


[component][list] no action on "next" button

The "Next" button is not implemented.

This bug is fixed


Play Function

Storybooks has this "play function" feature for writing inline tests for interactions.
Migrating to Ladle would require porting all the tests you've written with these functions to something new.
Instead it would be great if Ladle supported these by default.
Even better if it could be done better than storybook by running these tests in the background in an iframe or even a background process or something like that and not affect the window you're interacting with unless you choose to play the test manually.

Yea, I've been thinking a lot about this feature.
Internally, we have these foo.stories.test.js files that map stories into "play" functions. Also our play functions get expect and snapshot as params. The point is, we run this in our CI as a robust test suite and you can also run it locally in terminal. We built this before play functions were a thing.
I really like the idea of having this baked into the UI so it's easier to develop/debug/test but also want to preserve the "CI test suite" part that allows to do multiple assertions before/during/after the play function.

Even better if it could be done better than storybook by running these tests in the background in an iframe or even a background process or something like that and not affect the window you're interacting with unless you choose to play the test manually.

💯

I do think if you press "play" in the UI it should probably run it in the frame you see so that you can inspect it.

Probably not gonna support this type of "testing", it would be a lot to maintain. On the other hand if there is anything to make the integration with playwright and its test runner easier, always happy to support that.


LSI recomputation panpipes_clustering

Added the recomputation of the LSI for the ATAC modality to panpipes_clustering. Also added the dim_remove parameter only for ATAC and for both, LSI and PCA, the same way as in panpipes_preprocess: https://github.com/DendrouLab/panpipes/blob/main/panpipes/python_scripts/run_preprocess_atac.py

@SarahOuologuem can you check if filtering the atac layer by by hvg outputs the LSI dimred in the object? similarly to what mentioned in this issue https://github.com/DendrouLab/panpipes/issues/98
thanks!

@SarahOuologuem did you get a chance to check if filtering the atac layer by hvg outputs the LSI dimred in the object? i can merge when you're ready. thanks!!


Fix wrong arguments

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
2 unchanged lines in 2 files lost coverage.
Overall coverage remained the same at 23.077%

Files with Coverage Reduction
New Missed Lines
%

src/EventListener/SendMessageToTransportsEventListener.php
1
0%

src/Service/EnvelopeCollector.php
1
0%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
3

Relevant Lines:
13

💛  - Coveralls


Muting the remaining messages when show_progress=False

I was using the library and I found that when doing a cross validation, even when I chose the parameter show_progress to False, it showed some messages:
"Will train until cv error hasn't decreased in X rounds", in case of having early_stopping
"Stopping. Best iteration:\n[X] cv-mean:[...]\n"
I wrote some conditions for muting it when show_progress=Fase because if you run a bunch of models with a grid of parameters, the high number of messages can bother :D.

is everything ok?

Sorry for the delayed reply. Let us reuse the verbose_eval parameter

Ok I see,  I changed it! is it ok now?

can we merge it? what are the next steps? the tests are failing because it is a refactoring

please do a rebase to merge your commits, and it will be merged.http://xgboost.readthedocs.org/en/latest/dev-guide/contribute.html#how-to-combine-multiple-commits-into-one

done :D!

@ivallesp It seems that I merged the change too hastely and now the travis test break due to the change introduced by this PR, https://travis-ci.org/dmlc/xgboost/builds/110415766
Please make a PR to fix this issue, so the other changes can be savely merged in.

https://travis-ci.org/dmlc/xgboost/jobs/110415768#L974

@tqchen I just did it! Sorry but I made something wrong during the squash and the changes in the training file were not applied. I think it is solved now. Let's wait until tests done. Thank you for reporting it.

@tqchen yep! Now the tests passed and it is working correctly. Sorry for the inconvenience.


Kubernetes and Tectonic

I used the previous kubernetes install guide from CoreOS and its been working fine, but now when I go to the kubernetes docs on coreos.com it redirects to kubernetes+tectonic. Is there no longer a standalone kubernetes with coreos?

Hi @qrpike, those docs were unmaintained and out of date, and alternative resources are available. See this coreos-user thread for more details.


OpenCV version compatibility

Version
Comments

<IP_ADDRESS>
Recommended openCV version for PeekingDuck as of 15 Dec 2021. Doesn't work for M1 Mac.

From <IP_ADDRESS> to <IP_ADDRESS>
Not recommended to use these versions due to a few reported issues mainly due to QT issues with MacOS, such as imshow() not showing, or segmentation fault, or QT error in MacOS

<IP_ADDRESS>
Works with M1 Mac and the issues from earlier versions have been fixed by dropping QT support for Mac. Different behaviour for cv2.resize() and cv2.resize_window() when compared to <IP_ADDRESS>.

<IP_ADDRESS>
Coordinates need to be int instead of float for drawing

One correction: the int instead of float coordinates changes began with version 4.5.1.
The key difference between 4.5.1 and 4.5.2 is that the former auto typecast numpy.float32 (but not python float) into int while the latter does not. This is why PeekingDuck "works" with 4.5.1, but on 4.5.2 it shows the error for drawing coordinates to be int instead of float.


docs(README): add neotest-scala

Adds docs for neotest-scala adapter.
Here's a short video showing the adapter in action:
https://user-images.githubusercontent.com/6879030/191082489-fc8f9d2a-07f1-41de-ba2c-1efd81efb059.mov

Looks great, thanks for contributing! :smile:


pull v1.0 - manifest unkown

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

/kind feature

What happened:
trying to pull the v1.0 as states in the deploy.yml ended in
Error response from daemon: manifest for caicloud/event-exporter:v1.0.0 not found: manifest unknown: manifest unknown
i'm guessing you haven't release it yet..?
If that's the case - it the src buildable in master brach?
thanks!

We haven't released it. We've only tested in our dev environment. We will release it after using it for a while in production.
The master is ready and buildable, try REGISTRY=someregistry VERSION=v1.0.0 make container

The following image has been released:
caicloud/event-exporter:v1.0.0
caicloud/event-exporter:v1.0
caicloud/event-exporter:latest

/close


Crash on n2c for Cardano Testnet: chainsync loop failed BytesIndef

block e1ef53d961b282f380544071136b17992d63859d1ba92e02f25884161886c1f1
slot 58613322
thread '<unnamed>' panicked at 'chainsync loop failed: DownstreamError(Error { err: TypeMismatch(BytesIndef), pos: Some(6783), msg: "expected bytes (definite length)" })', /Users/sebastienguillemot/.cargo/registry/src/github.com-1ecc6299db9ec823/oura-1.4.3/src/sources/n2c/setup.rs:106:18

Oura version: 1.4.3
Pallas version: 0.10.0
cardano-node version: 1.34.1

Fixed on https://github.com/txpipe/pallas/pull/143 and released as pallas-primitives=0.10.1.
The issue was that the bytes variant of the plutus data struct accepts indef bytes (indef array of bytes) and the codec in Pallas wasn't supporting it.
Since it's a patch, downstream project should bump pallas-primitives to v0.10.1 by running cargo update -p pallas, change will only be reflected on the Cargo.lock file. I'll be doing a binary release of Oura to v1.4.4 but it won't have any effect on projects using Oura as a Cargo dependency.


Initialize LLVM AMDGPU Target

Without this initialization in-process compilation unable to create
a target machine and optimize bitcode properly.

Hmm...I guess we can turn inprocess back on by default with this.

Thank you!


Add MaterializedView relation type, update Snowflake adapter

Fix for issue https://github.com/fishtown-analytics/dbt/issues/1430
-- Adds MaterializedView to the list of RelationTypes in the BaseRelation class
-- Updates the snowflake__list_relations_without_caching macro to handle 'MATERIALIZED VIEW' table_type
I didn't find any unit tests for the BaseRelation parsing, so I just tested manually. Let me know if there's a good way to add a test for this.
Prior to the fix running a model in this project returned:
* Deprecation Warning: The adapter function `adapter.already_exists` is deprecated and will be removed in
 a future release of dbt. Please use `adapter.get_relation` instead.
 Documentation for get_relation can be found here:
 https://docs.getdbt.com/reference#adapter

Encountered an error:
Runtime Error
  Invalid arguments passed to "SnowflakeRelation" instance: type.'MATERIALIZED VIEW' is not one of ['table', 'view', 'cte', None]

After the fix the model runs successfully:
Running with dbt=0.13.0
* Deprecation Warning: The adapter function `adapter.already_exists` is deprecated and will be removed in
 a future release of dbt. Please use `adapter.get_relation` instead.
 Documentation for get_relation can be found here:
 https://docs.getdbt.com/reference#adapter

Found 140 models, 364 tests, 0 archives, 0 analyses, 103 macros, 0 operations, 0 seed files, 0 sources

09:31:09 | Concurrency: 3 threads (target='production_bi')
09:31:09 |
09:31:09 | 1 of 1 START table model bi.the_model................... [RUN]
09:31:17 | 1 of 1 OK created table model bi.the_model.............. [SUCCESS 1 in 8.01s]
09:31:17 |
09:31:17 | Finished running 1 table models in 14.96s.

Completed successfully

Done. PASS=1 ERROR=0 SKIP=0 TOTAL=1

Thanks @adriank-convoy - this looks great!
We're going to cut an 0.13.1 bugfix release relatively soon (within the next ~week) whereas 0.14.0 is going to take a little bit longer (5-6 weeks). If you're interested in getting this live for 0.13.1, you could apply your changed on top of dev/0.13.1 and then change the destination branch for this PR to dev/0.13.1 accordingly. If you're happy waiting for 0.14.0 to use this, then I think we can merge as-is once the automated tests pass

Not sure if I can change to source branch so I'm just going to re-do the pr from dev/0.13.1


Search Functionality Fails with Chinese Characters in Filenames or Paths

Thank you for developing this plugin.
During my testing, I noticed an issue with searching when filenames or folder paths contain Chinese characters. No results are found in such cases. However, when I remove the Chinese characters, the search works correctly.
For example:

Filename "你好.md": No text within this file can be searched.
Filename "hello.md": All content, including Chinese characters, can be searched.
I appreciate your attention to this matter.

After my testing, this problem has been fixed in v2.1.0 version, thank you very much


Add Building Docker Images using Google Jib

Google provides the Tool [Jib] (https://github.com/GoogleContainerTools/jib) to create Docker Images for Java Projects.
An example can be found here
One Advantage is that the resulting images are small in size and Jib utilizes caching to make the build process fast.
One disadvantage of this tool is that it encapsulates the whole process of creating a docker image, so one cannot see the "actual" Dockerfile that was created

done


the instruction for Docker implementation is abrupt

it suggests to run " following command" but there is no following command in the docker support readme file

could you extend the following, because the default build instructions won't work:
sh docker_run.sh
This will initialize docker. Clone this repository using following command and follow build and run instructions for ros2 package from here.

tried with
`mkdir ros2_ws
ros2_ws/src/
git clone https://github.com/NVIDIA-AI-IOT/ros2_trt_pose.git
rosdep install --from-paths src --ignore-src --rosdistro eloquent -y
colcon build
root@nvidia-desktop:/ros2__ws# colcon build
Starting >>> pose_msgs
Starting >>> ros2_trt_pose
--- stderr: pose_msgs
CMake Error at /usr/share/cmake-3.10/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
Could NOT find FastRTPS (missing: FastRTPS_INCLUDE_DIR FastRTPS_LIBRARIES)
Call Stack (most recent call first):
/usr/share/cmake-3.10/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
/opt/ros/eloquent/share/fastrtps_cmake_module/cmake/Modules/FindFastRTPS.cmake:95 (find_package_handle_standard_args)
/opt/ros/eloquent/share/rosidl_typesupport_fastrtps_c/cmake/rosidl_typesupport_fastrtps_c-extras.cmake:7 (find_package)
/opt/ros/eloquent/share/rosidl_typesupport_fastrtps_c/cmake/rosidl_typesupport_fastrtps_cConfig.cmake:38 (include)
/opt/ros/eloquent/share/rosidl_default_generators/cmake/rosidl_default_generators-extras.cmake:21 (find_package)
/opt/ros/eloquent/share/rosidl_default_generators/cmake/rosidl_default_generatorsConfig.cmake:38 (include)
CMakeLists.txt:23 (find_package)

Failed   <<< pose_msgs [0.82s, exited with code 1]
Aborted  <<< ros2_trt_pose [1.38s]   `

passing the colcon build with
CMAKE_PREFIX_PATH=$AMENT_PREFIX_PATH:$CMAKE_PREFIX_PATH


UI weirdness

Bob gets a crash when he deletes the value from the cell size parameter in the UI but Mona doesn't get the crash for the same action!
Here's the crash output:

And here's app.R line 230 from dev branch 0b1b5185351a588af25ecdbf0e469f77a4003346

I was able to fix the bug by adding an as.numeric() test on the cell size. Specifically, replacing this ...
if (! isEmpty (gps$original) && input$cellsize >= 1)
... with this ...
if (! isEmpty (gps$original) && is.numeric(input$cellsize) && input$cellsize >= 1)
After making this change, the shiny app works fine and actually prohibits me from entering something that's not a number.
Regarding the source of the error, I'm wondering if this is a Windows vs. Mac problem (I'm on a Mac).

On a related note, I noticed that you're only testing the cell size before enabling the run button (runx). We should probably add tests to make sure that the other values are valid

sig2obs is numeric; sig2obs > 0
tmax is numeric; tmax > 0
buffer is numeric; buffer ≥0

Finally, since R does short circuiting (see https://r-critique.com/short-circuit-evaluation), I would recommend putting the is.numeric() tests before the comparisons.

Implemented changes and committed to the dev branch

Closed


Allow to disable it for specific role

Hi.
Great module!
is there a way to hide it on specific roles?

You can use a custom hook to intercept the settings before they are returned to the app


Update DataManager.java

this add the same data twice and then distinct it

Can one of the admins verify this patch?

Hi, that was implemented that way so data from the cache (SharedPreferences) is returned first, so the app can display something while is loading the server data. The distinct() prevents from emitting the same data twice for those cases that the server data and the cached data is exactly the same.


Registry API

Makeup should support a global registry in which lexers can register themselves on application start.
That way a new lexer can be made available from a language name just by declaring it as a dependency.

Motivation here: https://github.com/elixir-lang/ex_doc/pull/853#issuecomment-390621535

Pinging @josevalim because he was the one to suggest this in an ExDoc issue.
The API I have is very simple. Makeup now executes code on application start, which creates a lexer registry where you can register lexers by name.
Lexers can make use of it by registering themselves on application start. For example, the code I've written for the Elixir lexer:
defmodule Makeup.Lexers.ElixirLexer.Application do
  @moduledoc false
  use Application

  alias Makeup.Registry
  alias Makeup.Lexers.ElixirLexer

  def start(_type, _args) do
    Registry.register_lexer_with_name("elixir", {ElixirLexer, []})
    # Use the normal Elixir lexer for `iex` prompts
    Registry.register_lexer_with_name("iex", {ElixirLexer, []})
    Supervisor.start_link([], strategy: :one_for_one)
  end
end

This could be made even more concise with something like:
defmodule Makeup.Lexers.ElixirLexer.Application do
  use Makeup.Registry, names: ["elixir", "iex"]
end

but that seems needlessly obscure.
Should I add an extension registry, so that the Elixir lexer could be associated with the .ex and .exs extensions, for example?

It looks great.

Should I add an extension registry, so that the Elixir lexer could be associated with the .ex and .exs extensions, for example?

Yes. Probably the same registry is fine. :+1:

Done since version 0.8


UI Enhancement

Fixes #41 ,
1.Color of Save button has been changed to #E8EAF6 (Very light with a shade of Indigo), when disabled:

2.Tick icon is added in dafault in clock button:

Along with the course code also display the course name in upcoming

Updated

@neeraj-2 never resolve a conversation on your own.
ie if you are the author you should nor resolve a conversation on your own

Updated

@neeraj-2 never resolve a conversation on your own.
ie if you are the author you should nor resolve a conversation on your own

I tried ,but it seems a little messy

Along with the course code also display the course name in upcoming

I tried ,but it seems a little messy:

I tried ,but it seems a little messy:

@neeraj-2
You can display fix number of character then truncate the string and add ... in last

I tried ,but it seems a little messy:

@neeraj-2
You can display fix number of character then truncate the string and add ... in last

Done,now it looks as shown:


Trim unnecessary whitespaces between a macro call and a semicolon

Closes #2727.

Thanks for the review, addressed the comment.


phone-number: Correct test to more-than check vs equal-to check

Fix #608

Thanks for the heads-up @petertseng (and the title help @kotp).
Just to note that I did take a quick glance at the CONTRBUTING.md but found it very long to try and find the relevant information in (i.e. the bit Peter linked)... Maybe that, or something like it, could be split out to the same directory and linked to to make it easier to find? Just a suggestion.


Fix ios font family error

Findings:

The assets folder needs to be refactored from ./src/assets to ./assets. In order to link these files in the native Resource folder of ios
The Modak-Regulat.ttf font file is not supported by the Mac ecosystem. Verified by the font-book app.


Performance of "buffered" bitmaps is 10x worse

We can expect a small performance difference between bitmaps that are backed by an array, and bitmaps that are backed by a java.nio.ByteBuffer instance. However, the current performance difference is huge and probably unacceptable.
I suspect that much of the blame can be put on the getWords method which essentially copies the entier content to an array:
https://github.com/lemire/javaewah/blob/master/src/main/java/com/googlecode/javaewah/LongBufferWrapper.java#L50
Obviously, we would want to avoid such copies if at all possible.
Note that the swap method is similarly sluggish:
https://github.com/lemire/javaewah/blob/master/src/main/java/com/googlecode/javaewah/LongBufferWrapper.java#L151
though it probably does not matter as much.
Here are my results:
        PerformanceTest.cardinality: [measured 3 out of 6 rounds, threads: 1 (sequential)]
         round: 0.49 [+- 0.01], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 0, GC.time: 0.00, time.total: 3.41, time.warmup: 1.94, time.bench: 1.47
        PerformanceTest.toarray: [measured 3 out of 6 rounds, threads: 1 (sequential)]
         round: 1.52 [+- 0.05], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 12, GC.time: 0.01, time.total: 9.15, time.warmup: 4.61, time.bench: 4.55
        PerformanceTest.bigunion: [measured 3 out of 6 rounds, threads: 1 (sequential)]
         round: 0.57 [+- 0.01], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 5, GC.time: 0.00, time.total: 3.52, time.warmup: 1.80, time.bench: 1.72
        PerformanceTest.bigunionbuf: [measured 3 out of 6 rounds, threads: 1 (sequential)]
         round: 2.85 [+- 0.06], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 16, GC.time: 0.01, time.total: 17.34, time.warmup: 8.78, time.bench: 8.56
        PerformanceTest.toarraybuf: [measured 3 out of 6 rounds, threads: 1 (sequential)]
         round: 11.80 [+- 0.04], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 118, GC.time: 0.07, time.total: 69.53, time.warmup: 34.14, time.bench: 35.39
        PerformanceTest.cardinalitybuf: [measured 3 out of 6 rounds, threads: 1 (sequential)]
         round: 5.05 [+- 1.15], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 1, GC.time: 0.00, time.total: 28.06, time.warmup: 12.91, time.bench: 15.15

Unfortunately, the getWords method is not called in the benchmarks computing the bitmap cardinality and converting the bitmap to an array. So it cannot explain the poor performance.
Still, I just removed this method.

Unfortunately, the getWords method is not called in the benchmarks computing the bitmap cardinality and converting the bitmap to an array. So it cannot explain the poor performance.
Still, I just removed this method.

I've noticed a performance difference when I change the backend buffer from a bytebuffer to a longbuffer. Just replace the method convertToMappedBitmap with :
public static EWAHCompressedBitmap convertToMappedBitmap(EWAHCompressedBitmap orig) throws IOException {
        final EWAHCompressedBitmap bitmap = new EWAHCompressedBitmap(LongBuffer.wrap(new long[orig.sizeInBytes()]));
        for (int pos : orig) {
            bitmap.set(pos);
        }
        return bitmap;
    }

Here are the results I get (with openjdk/jdk1.8.0_40):
PerformanceTest.bigunion: [measured 10 out of 13 rounds, threads: 1 (sequential)]
 round: 0.54 [+- 0.04], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 1, GC.time: 0.09, time.total: 7.24, time.warmup: 1.89, time.bench: 5.36
PerformanceTest.bigunionbuf: [measured 10 out of 13 rounds, threads: 1 (sequential)]
 round: 1.14 [+- 0.08], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 2, GC.time: 0.04, time.total: 14.86, time.warmup: 3.47, time.bench: 11.39
PerformanceTest.toarray: [measured 10 out of 13 rounds, threads: 1 (sequential)]
 round: 1.90 [+- 0.03], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 10, GC.time: 0.02, time.total: 24.96, time.warmup: 5.96, time.bench: 19.00
PerformanceTest.toarraybuf: [measured 10 out of 13 rounds, threads: 1 (sequential)]
 round: 3.36 [+- 0.03], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 8, GC.time: 0.02, time.total: 43.77, time.warmup: 10.12, time.bench: 33.65
PerformanceTest.cardinalitybuf: [measured 10 out of 13 rounds, threads: 1 (sequential)]
 round: 1.40 [+- 0.01], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 0, GC.time: 0.00, time.total: 18.25, time.warmup: 4.24, time.bench: 14.00
PerformanceTest.cardinality: [measured 10 out of 13 rounds, threads: 1 (sequential)]
 round: 0.91 [+- 0.03], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 0, GC.time: 0.00, time.total: 11.84, time.warmup: 2.70, time.bench: 9.14

As you can see, the overhead factor here is ~2. So there is a cost to work over a bytebuffer wrapped by a longbuffer.
To explain this 2x factor, I believe it is due to additional bounds checking done by implementations of LongBuffer like HeapLongBuffer when you access a word at a given position.

I've noticed a performance difference when I change the backend buffer from a bytebuffer to a longbuffer. Just replace the method convertToMappedBitmap with :
public static EWAHCompressedBitmap convertToMappedBitmap(EWAHCompressedBitmap orig) throws IOException {
        final EWAHCompressedBitmap bitmap = new EWAHCompressedBitmap(LongBuffer.wrap(new long[orig.sizeInBytes()]));
        for (int pos : orig) {
            bitmap.set(pos);
        }
        return bitmap;
    }

Here are the results I get (with openjdk/jdk1.8.0_40):
PerformanceTest.bigunion: [measured 10 out of 13 rounds, threads: 1 (sequential)]
 round: 0.54 [+- 0.04], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 1, GC.time: 0.09, time.total: 7.24, time.warmup: 1.89, time.bench: 5.36
PerformanceTest.bigunionbuf: [measured 10 out of 13 rounds, threads: 1 (sequential)]
 round: 1.14 [+- 0.08], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 2, GC.time: 0.04, time.total: 14.86, time.warmup: 3.47, time.bench: 11.39
PerformanceTest.toarray: [measured 10 out of 13 rounds, threads: 1 (sequential)]
 round: 1.90 [+- 0.03], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 10, GC.time: 0.02, time.total: 24.96, time.warmup: 5.96, time.bench: 19.00
PerformanceTest.toarraybuf: [measured 10 out of 13 rounds, threads: 1 (sequential)]
 round: 3.36 [+- 0.03], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 8, GC.time: 0.02, time.total: 43.77, time.warmup: 10.12, time.bench: 33.65
PerformanceTest.cardinalitybuf: [measured 10 out of 13 rounds, threads: 1 (sequential)]
 round: 1.40 [+- 0.01], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 0, GC.time: 0.00, time.total: 18.25, time.warmup: 4.24, time.bench: 14.00
PerformanceTest.cardinality: [measured 10 out of 13 rounds, threads: 1 (sequential)]
 round: 0.91 [+- 0.03], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 0, GC.time: 0.00, time.total: 11.84, time.warmup: 2.70, time.bench: 9.14

As you can see, the overhead factor here is ~2. So there is a cost to work over a bytebuffer wrapped by a longbuffer.
To explain this 2x factor, I believe it is due to additional bounds checking done by implementations of LongBuffer like HeapLongBuffer when you access a word at a given position.

@gssiyankai
In practice, when doing memory-file mapping, we are going to work from a ByteBuffer. It is how the Java API works.
Cheers!

@gssiyankai
In practice, when doing memory-file mapping, we are going to work from a ByteBuffer. It is how the Java API works.
Cheers!

We should change the benchmark so that the mapped bitmaps are actually on disk... this would give us the most realistic test possible.

We should change the benchmark so that the mapped bitmaps are actually on disk... this would give us the most realistic test possible.

@lemire
Benchmarks updated.
Results are better, but the 2x factor remains...
Best !

@lemire
Benchmarks updated.
Results are better, but the 2x factor remains...
Best !

@gssiyankai Excellent. We should take some time to study this 2x factor, to make sure it cannot be improved.

Do you have an idea of the overhead with roaring bitmap ?

It is a valid question. Let me find out.

Ok. I have implemented the exact same benchmark with Roaring, see there:
https://github.com/lemire/RoaringBitmap/blob/master/src/test/java/org/roaringbitmap/buffer/TestBenchmarkBasic.java
Assuming that you run the benchmarks on the same machine, the results should be comparable.
Here are the results, on the same machine (recent haswell processor, recent OpenJDK):
    $ java -version
    openjdk version "1.8.0-jdk8u60-b00"
    OpenJDK Runtime Environment (build 1.8.0-jdk8u60-b00-20141223)
    OpenJDK 64-Bit Server VM (build 25.40-b23, mixed mode)

    $ git clone https://github.com/lemire/RoaringBitmap.git
    $ cd RoaringBitmap
    $ mvn -Dtest=TestBenchmarkBasic test
    Running org.roaringbitmap.TestBenchmarkBasic
    TestBenchmarkBasic.bigunion: [measured 10 out of 13 rounds, threads: 1 (sequential)]
     round: 0.47 [+- 0.01], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 1, GC.time: 0.00, time.total: 6.26, time.warmup: 1.51, time.bench: 4.75

    Running org.roaringbitmap.buffer.TestBenchmarkBasic
    TestBenchmarkBasic.bigunionbuf: [measured 10 out of 13 rounds, threads: 1 (sequential)]
     round: 0.92 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 3, GC.time: 0.00, time.total: 12.12, time.warmup: 2.90, time.bench: 9.21
    TestBenchmarkBasic.bigunion: [measured 10 out of 13 rounds, threads: 1 (sequential)]
     round: 0.64 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 2, GC.time: 0.00, time.total: 8.39, time.warmup: 1.98, time.bench: 6.41

    $ git clone https://github.com/lemire/javaewah.git
    $ cd javaewah
    $ mvn -Dtest=PerformanceTest test
    Running com.googlecode.javaewah32.PerformanceTest
    PerformanceTest.bigunion: [measured 3 out of 6 rounds, threads: 1 (sequential)]
     round: 0.44 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 2, GC.time: 0.00, time.total: 2.69, time.warmup: 1.38, time.bench: 1.31
    PerformanceTest.bigunionbuf: [measured 3 out of 6 rounds, threads: 1 (sequential)]
     round: 0.78 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 2, GC.time: 0.00, time.total: 4.73, time.warmup: 2.39, time.bench: 2.34

    Running com.googlecode.javaewah.PerformanceTest
    PerformanceTest.bigunion: [measured 3 out of 6 rounds, threads: 1 (sequential)]
     round: 0.46 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 6, GC.time: 0.00, time.total: 2.81, time.warmup: 1.43, time.bench: 1.38
    PerformanceTest.bigunionbuf: [measured 3 out of 6 rounds, threads: 1 (sequential)]
     round: 0.86 [+- 0.00], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 7, GC.time: 0.00, time.total: 5.20, time.warmup: 2.63, time.bench: 2.58

@gssiyankai
What these results suggest to me is that our performance is good. That is, we get the same kind of degradation as Roaring.
I would suggest we close this issue and the release JavaEWAH 1.0.0 (yes!). If you have more stuff to get in before this major release, please do so. I have a few irrelevant things to tune out before the release...

There's been some discussions about bound checking:
https://groups.google.com/forum/#!topic/mechanical-sympathy/UD0cPwMvNEY
http://stackoverflow.com/questions/11174231/compare-direct-and-non-direct-bytebuffer-get-put-operations
Looks like the best comprise would be to make the LongBufferWrapper work on a long array and update the java.nio.LongBuffer when necessary.

@lemire
I have the clear feature that is ready. I've been working on it some weeks ago. I can push it to the project repo by the end of the week.

Looks like the best comprise would be to make the LongBufferWrapper work on a "small" long array and update the java.nio.LongBuffer when necessary

I fail to see how this helps. You might as well deserialize the bitmap, work on it and then re-serialize it as needed. But then you end having to manage your collection of bitmaps in Java RAM: when RAM gets exhausted, you have to release some of the bitmaps, and so on.
The purpose of the memory-file mapping is that you get to avoid this management and let the OS do it for you (and the OS can do it much better in some cases).

@gssiyankai
We might be able to use Unsafe "safely" with this library:
https://github.com/alexkasko/unsafe-tools
Or we could adapt it for our needs.

Looks like the best comprise would be to make the LongBufferWrapper work on a "small" long array and update the java.nio.LongBuffer when necessary

I fail to see how this helps. You might as well deserialize the bitmap, work on it and then re-serialize it as needed. But then you end having to manage your collection of bitmaps in Java RAM: when RAM gets exhausted, you have to release some of the bitmaps, and so on.

The purpose of the memory-file mapping is that you get to avoid this management and let the OS do it for you (and the OS can do it much better in some cases).

I hope I'll be clearer. What I meant is to use a fixed-size long array, for example a long array of size 10, that will act like a sliding window on the java.nio.LongBuffer. If I know I'm going to read the 10 first words, I can fetch all of them in a one call and copy them in the array at the cost of one bound check instead of 10. But this solution requires more memory (depending on the size of the array) and some changes in all algorithms.
Anyway, forget about this at the moment. As you suggested, let's close the issue and release since javaewah is on par with roaring.

@gssiyankai
Right. The rolling window makes logical sense, but you replace the bound checks with a copy and some other overhead. It is unclear to me whether you will come ahead.
For now, I am closing this particular issue since we agree that the performance is now acceptable.
Eventually, however, we should fight back against this 2x factor.


How to create thumbnail of Wowza Videos?

Hi,
I am looking for creating the thumbnails for all the videos which we have uploaded on Wowza.
I also want to know can we save these thumbnails on our server end, if it is possible to create thumbnails with the plugin.
I found this source link ( https://www.wowza.com/docs/how-to-generate-thumbnail-images-of-live-streams ), but this code is in JAVA. which we are not familiar with.
Need to do the same with this module.
Let me know if this is available or not?
Or is there any other way rather having java.
Thanks,

I would suggest asking Wowza support if they have JavaScript code for achieving something similar.


Unkown calls to offers endpoint

I don't know if this is a bug or not or if it's coming from the SDK. I'm using 3.35.1 in direct mode and I have a serverless account setup. When I look at the Insights page for the account I'm seeing requests being made every hour or so (sometimes 2 hours, sometimes a longer break). These are not calls coming from our app, there actually isn't any activity from our app. When I drill down into the logs it appears something is making calls to the offers endpoint. For Ex:
OperationName: Update
ConnetionMode: Gateway
UserAgent: MasterService
RequestResourceId: offers/VwPH
ClientIpAddress: <IP_ADDRESS>
Does anyone know where this is coming from and why?
I created a ticket in Azure for this but they've not been much help.

Can you please provide more details on this? May be a screenshot of what you are seeing?
May I know what's the concern here, you just want to know for your information, or you are seeing any misbehavior by SDK due to these calls?

I don't know where they are coming from. I didn't think it was the SDK but wanted to make sure. Must be something with the cosmos service itself, but Azure support has been useless to this point in answering it.

@festivus Can you please share your support case number privately? My email is maquaran{at}microsoft.com. I'll see how we can streamline the support. I'm sorry about this support experience.


Improvements

This pull request removes babel completely and uses tsc and uglifyjs only.
Adds also linting before commiting and prettier integration.
Also I switched to yarn and added missing package typescript.
I could add full webpack integration but for that simple code, you dont need it.
That's how the package looks like after (If I forgot some files, please tell me):
npm notice 📦<EMAIL_ADDRESS>npm notice === Tarball Contents ===
npm notice 1.4kB package.json
npm notice 1.1kB LICENSE
npm notice 3.8kB README.md
npm notice 120B  lib/index.d.ts
npm notice 598B  lib/index.js
npm notice === Tarball Details ===
npm notice name:          react-use-localstorage
npm notice version:       2.4.0
npm notice filename:      react-use-localstorage-2.4.0.tgz
npm notice package size:  3.2 kB
npm notice unpacked size: 7.0 kB
npm notice shasum:        c7c7b429afe66a01ed91370065da2d45f9f4952d
npm notice integrity:     sha512-PUOhUhaGfacMx[...]P5QCpwh6IyYEQ==
npm notice total files:   5

Also UglifyJS has more features if you need like manging, compressing etc.
Here is a example after compression enabled:
"use strict";var __importDefault=this&&this.__importDefault||function(mod){return mod&&mod.__esModule?mod:{default:mod}};Object.defineProperty(exports,"__esModule",{value:!0});var react_1=__importDefault(require("react"));function useLocalStorage(key,initialValue){void 0===initialValue&&(initialValue="");var _a=react_1.default.useState(function(){var value=localStorage.getItem(key)||initialValue;return localStorage.setItem(key,value),value}),item=_a[0],setValue=_a[1];return[item,function(newValue){setValue(newValue),window.localStorage.setItem(key,newValue)}]}exports.default=useLocalStorage;

The declaration file looks like this now:
import { Dispatch } from 'react';
export default function useLocalStorage(key: string, initialValue?: string): [string, Dispatch<string>];

Do not merge this yet :)

That can be merged now if you agree.

Thank you @TheAifam5.
I have played around with this PR and looks good & will merge.
(And learned a good flow [dropping babel]).
Using husky for the pre-commit hook idea was awesome
(I just learned of it from Kent C. Dodds' video on Egghead yesterday) 🙂.
As a side question,
May I ask about the benefit of using yarn over npm?

@allcontributors please add @TheAifam5 for code, infra

@TheAifam5 I've bumped the major version as Dispatch<string> is a breaking change.
https://www.npmjs.com/package/react-use-localstorage/v/3.0.0

So IMHO yarn can handle dependecies better than npm. The folder is much smaller and Yarn has a lot of features like “resolution” or “workspaces” if you will plan do a monorepo with multille packages.
Mainly the reason of switching to Yarn was the performance. There is also NPX but I didn’t tested it yet.
Npm since I last used it has a lot of features too, some of them are just ripped of from yarn.

Babel is really nice and you can integrate it with Typescript code just by installing preset @babel/preset-typescript. The one issue I see there, does not generate .d.ts files.
That could also solve by:
Using babel with Typescript preset.
Copy index.ts to output for Typescript users.
Publishing the source code in package allows the user to decide how the package should be transpiled, for JavaScript users there would be pretranspiled JS file already available.

Thanks for the great explanation @TheAifam5 👍. Learned much 🙂
For the .d.ts file the previous implementation generated one using tsc while .ts files were transipiled using babel, which I didn't think was ideal.


Identifiers validation failed for dataset Colecta de Physalis (Solanaceae) al norte del Trópico de Cáncer

Identifier validation failed for the dataset Colecta de Physalis (Solanaceae) al norte del Trópico de Cáncer:

Crawler attempt: 4
Publishing organization: Comisión nacional para el conocimiento y uso de la biodiversidad
Publishing country: MEXICO
Installation: IPTCONABIO
Number of occurrences indexed for this dataset at the time of the issue: 712
Cause: GBIF ID problems exceed 50% threshold: 100% duplicates; 712 total records; 712 absent records
Pipelines execution steps: http://api.gbif.org/v1/pipelines/history/execution/3757083/step

New IDs sample:

Old IDs sample:

urn:catalog:FCN-UAQ:QMEX:12775:b1d682be3ff12a739818987830a2951c
urn:catalog:FCN-UAQ:QMEX:12776:ee90fd5a2be56acc47194f9e39641285
urn:catalog:FCN-UAQ:QMEX:12777:3076fb418c32204d18369881abbc1d4b
urn:catalog:FCN-UAQ:QMEX:12778:59a8de1c89e12cdad6e3a48eaa26f643
urn:catalog:FCN-UAQ:QMEX:12779:bc338a22999210f3683c91729ec76b39
urn:catalog:FCN-UAQ:QMEX:12780:45dca07fc8c186020b26cd11e2fc86c1
urn:catalog:FCN-UAQ:QMEX:12781:4c8c3fa66235f7e34f109e59e8cba34d
urn:catalog:FCN-UAQ:QMEX:12782:f808b8e2eb67acf4a0d3682b472130ca
urn:catalog:FCN-UAQ:QMEX:12783:7693d4535209d4dbdce02a4d1b372a2c
urn:catalog:FCN-UAQ:QMEX:12784:491317327261c85b581972c917146df8

Publisher email
Hello,
I am contacting you from the GBIF Secretariat about a dataset published by the Colecta de Physalis (Solanaceae) al norte del Trópico de Cáncer : https://doi.org/10.15468/gjrz44.
We noticed that the occurrenceIDs were changed. We have temporarily paused the ingestions of this dataset.
As you might already know, when an occurrence record has a new occurrenceID for a given dataset, our system considers it to be a new occurrence.
This means that it will be given a new gbifid and a new occurrence URL (like this one: https://www.gbif.org/occurrence/1252968762) and the old gbifid and URL will be deprecated.
In this case, this means that the occurrence URLs would be deprecated when ingesting the newest versions of these datasets.
We would like to check with you if those changes were intentional. Do you know if this is the case?
Please let us know, thanks! We are happy to resume the dataset ingestion.
Note that some users rely on those occurrence URLs and gbifids (like https://bionomia.net for example). In an attempt to improve
the stability of the occurrence URLs and gbifids, we have implemented a warning system to detect these type of changes in datasets (see this news item).
If the data publisher can provide us with a list of old and new occurrenceIDs per record, we can avoid the identifier and URL changes. Could that be an option?
Please let us know if you have any question. Thanks!
All the best,

You can skip/fix identifier validation using the registry UI.

Identifier validation still  failing for the dataset Colecta de Physalis (Solanaceae) al norte del Trópico de Cáncer:

Crawler attempt: 5
Publishing organization: Comisión nacional para el conocimiento y uso de la biodiversidad
Publishing country: MEXICO
Installation: IPTCONABIO
Number of occurrences indexed for this dataset at the time of the issue: 712
Cause: GBIF ID problems exceed 50% threshold: 100% duplicates; 712 total records; 712 absent records
Pipelines execution steps: http://api.gbif.org/v1/pipelines/history/execution/3757203/step

New IDs sample:

Old IDs sample:

urn:catalog:FCN-UAQ:QMEX:12775:b1d682be3ff12a739818987830a2951c
urn:catalog:FCN-UAQ:QMEX:12776:ee90fd5a2be56acc47194f9e39641285
urn:catalog:FCN-UAQ:QMEX:12777:3076fb418c32204d18369881abbc1d4b
urn:catalog:FCN-UAQ:QMEX:12778:59a8de1c89e12cdad6e3a48eaa26f643
urn:catalog:FCN-UAQ:QMEX:12779:bc338a22999210f3683c91729ec76b39
urn:catalog:FCN-UAQ:QMEX:12780:45dca07fc8c186020b26cd11e2fc86c1
urn:catalog:FCN-UAQ:QMEX:12781:4c8c3fa66235f7e34f109e59e8cba34d
urn:catalog:FCN-UAQ:QMEX:12782:f808b8e2eb67acf4a0d3682b472130ca
urn:catalog:FCN-UAQ:QMEX:12783:7693d4535209d4dbdce02a4d1b372a2c
urn:catalog:FCN-UAQ:QMEX:12784:491317327261c85b581972c917146df8

Publisher email
Hello,
I am contacting you from the GBIF Secretariat about a dataset published by the Colecta de Physalis (Solanaceae) al norte del Trópico de Cáncer : https://doi.org/10.15468/gjrz44.
We noticed that the occurrenceIDs were changed. We have temporarily paused the ingestions of this dataset.
As you might already know, when an occurrence record has a new occurrenceID for a given dataset, our system considers it to be a new occurrence.
This means that it will be given a new gbifid and a new occurrence URL (like this one: https://www.gbif.org/occurrence/1252968762) and the old gbifid and URL will be deprecated.
In this case, this means that the occurrence URLs would be deprecated when ingesting the newest versions of these datasets.
We would like to check with you if those changes were intentional. Do you know if this is the case?
Please let us know, thanks! We are happy to resume the dataset ingestion.
Note that some users rely on those occurrence URLs and gbifids (like https://bionomia.net for example). In an attempt to improve
the stability of the occurrence URLs and gbifids, we have implemented a warning system to detect these type of changes in datasets (see this news item).
If the data publisher can provide us with a list of old and new occurrenceIDs per record, we can avoid the identifier and URL changes. Could that be an option?
Please let us know if you have any question. Thanks!
All the best,

You can skip/fix identifier validation using the registry UI.


feat(sdk): update Thanos Sepolia addresses

Update the Thanos Sepolia configuration for our new testnet on Sepolia and release it.

Thank you, Nam. After testing the deposit is working fine. But Withdrawal seems to have some issues. Have you checked the "prove the message" and "resolving the dispute game" cases for L2 native token, ETH, and ERC20 assets?

Thank you, Nam. After testing the deposit is working fine. But Withdrawal seems to have some issues. Have you checked the "prove the message" and "resolving the dispute game" cases for L2 native token, ETH, and ERC20 assets?

There is not issue in the SDK. This issue is related the network configuration.


Data Dictionary

We currently have a survey up for potential tool users to provide us with information on their data priorities, and will have results of this survey before the design meeting.
For each of these data categories, we want to also provide a list of specific data sources, download and save the specific source we'll use, and do a quick assessment of what is contained in the data. We'll want to summarize this in a data dictionary, which contains:

1-2 sentence summary of what is in the data source
Highlight key attributes that are of the most use to this project
Flag any potential limitations or difficulties of using the data
Copy or link to a full description of the data attributes, if available.

Starter list of data categories is here, but ask Neal for the latest updates on the data we have already obtained:
https://github.com/codefordc/housing-insights/wiki/Data-Sources
Related tickets:
#16 , #7

Framework for Data Dictionary currently up in Google Doc Format
Plan of action is to:

Complete doc for all tables on postgres
Convert to Markdown tables
Post to housing-insights wiki pages

Hi @lizmeister321 and @salomoneb  - hope your Thanksgiving was good! What's your plan for working on this? Do you think you'll be able to do some this week? I'd like to make sure we have a good, digestable summary version that we can print and have on hand at our Wednesday hacknight, so that everyone has something to refer to about the data we have available. It doesn't have to be the complete data dictionary, but we should have the core structure.

And let me know what help you need!

Hey @NealHumphrey - I was planning to dive into this over the next few nights. I'll probably want to defer to/speak with Liz about this since she set up the Google doc, but for the short term, I'd like to:
A) Add each column field for each database table in Column A of the corresponding Google sheet;
B) Define each field in Column B;
C) Add in URL references/citations for each table and/or specific data fields where appropriate. We'll be able to more specifically determine what this involves as we move through the data.
@lizmeister321 - If this plan works for you (or if you have other ideas), maybe we can talk on Slack and figure out how to divide everything up?

Great, sounds good. Most of the data has data dictionaries you can copy over - but often these are not quite as descriptive as we would like.
I would suggest aiming for a rough version of your 3 steps based on primarily copy-pasting from source dictionaries, and then add a 3rd step of pulling out or summarizing what look like might be the most important fields  - it doesn't have to be perfect, just a best first guess. That way people can read just the summary before making mockups, and look at the full list if they need to know if a specific field is available.

That sounds good to me. I agree with your idea about calling out the most valuable fields. I'll think about the best way to show that information.

@salomoneb Looks like a great start on the data dictionary in the Google Docs file. I just wanted to check in and make sure I can see it all - is this everything you've done so far?
Next steps:

I can copy in stuff related to the Preservation Catalog that I have handy on the appropriate tabs.

We need a 1-2 page summary of key things available in the data, for people to reference when drawing mockups.

Do you think you'll be able to do anything on 2 tonight or tomorrow? If not, I can plan to work on it to make sure we have materials ready for Wednesday.

@NealHumphrey Yeah, that's it for now. Wading through the census data/various census websites and tools took a bit longer than planned. I was going to wrap up the tax info tonight and start on another table. I spoke to @lizmeister321 the other day and she was going to work on the neighborhoodinfodc info.
I don't know if I'll be able to finish the other tables and do a 1-2 page summary before Wednesday (though maybe!), but I can probably do one or the other. What's most helpful/highest priority for you? Seems like the summary is the goal, but we need the definitions first. How about I see how much I can get done tonight and then we can reassess where we are tomorrow night?

Great, sounds good. I am teaching tonight, but can do some work on this during the daytime tomorrow. What if you send me an update on wherever you land this evening. I can do a first draft of summary during the day tomorrow, and then maybe you can just do a quick pass and add anything you think is necessary?

Sounds good. I'll send something later.

Filled in everything except the neighborhoodinfo tables. I wanted to get these as well, but Excel 2011 for Mac apparently can't open XML spreadsheet files. I might be able to add tomorrow when I'm on my PC work computer.

Just about every cell that's shaded has a comment attached to it.
Green cells are informational notes.
Orange cells are issues (not clear on the field definition, data might be inconsistent or missing)
The overview tab just has noteworthy links for now, but we can flesh this out.

I'm going to think a little bit more about the analysis portion tomorrow.

I updated the data dictionary with the census mapping and building permit tables. The building permit data doesn't have field descriptions, but I think it's still important to list the fields. We can come up with our own descriptions later if necessary.
I also created a new wiki page for general PostGres queries. It might seem sort of trivial, but I always appreciate having these things handy. People should feel free to contribute to this.

I'm moving the link to the data dictionary onto the housinginsights.org/resources page. We can keep it up to date as we go. Closing this issue


"The validation error was 'authors may not be empty'"

You have one or more invalid gemspecs that need to be fixed.
The gemspec at gems/spree-search-sku-3933a65b52e5/spree_search_sku.gemspec is not valid. Please fix this gemspec.
The validation error was 'authors may not be empty'

I'm guessing the gem spec format has changed since this gem was made.
This is using rails 5.0.1 and ruby 2.4.0

Ok the author line is just commented out so bringing it back lets you install the gem again however there are a lot of other bits in this gem that don't work with a modern spree version.


Add first v2.3 RC

LGTM - one minor comment about ordering of the versions file (I think 2.3 should be at the top)


[Feature]:  Documentation and example of ${envvar} sweep command configuration usage

Description
The ${envvar} macro in the sweep command configuration has been a very useful feature, but I found the documentation/macro description misleading and wished there was an example of how to use it.
Suggested Solution
Update macro description
Currently, the `${envvar}$ macro description is as follows:

However, this should be corrected to

A way to pass environment variables. ${envvar:MYENVVAR} expands to the value of MYENVVAR environment variable.

The bolded text highlights the change: no space between colon and MYENVVAR. This requirement becomes evident in how the command is parsed (i.e. by the agent's envvar regex object).
Example usage
The follow example demonstrates how the environmental variable macro can be used on its own and with other macros:
command:
  - ${env}
  - ${interpreter}
  - ${envvar:VAR1}
  - ${program}
  - "--param=${envvar:VAR2}"
  - ${args}

where, for example, VAR1="-m cProfile" is a variable to modify which profiler, and VAR2 is a runtime variable.
Alternatives
No response
Additional Context
No response

Hi @ezhang94, thanks for reporting this! You're right that this needs to be ${envvar:MYENVVAR} so will ping our team internally to update this. Thanks!

Hi @ezhang94, this has been updated on our docs. Thanks!


Select Inputs

At the moment options for select input fields only accepts an array of values. It would be nice if we could also provide an array of objects, so that the visible option is different from its value.
[
  {
    option: 'Germany'
    value: 'de'
  }
  {
    option: 'France'
    value: 'fr'
  }
]

Yeah, this is in the roadmap. The way it will works is that you give it a collection of objects and you specify which property is going to be the value and which will be the display.
Something like:
    <select data-bind="options: availableCountries,
                       text: 'countryName',
                       value: selectedCountry"></select>

Check out the options documentation -> Object Array: http://viewmodel.meteor.com/#optionsBk

btw, that's with ViewModel version 1.6.1


MODCLUSTER-479 Add support for Tomcat 9

Jira
https://issues.jboss.org/browse/MODCLUSTER-479

Triggering build using a merge of 186d224ccbd2f5305453573a87f519cfab58c633 on branch master:
Private: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/mod_cluster-1.3.x-pull-player-executor/

Build 176 outcome was FAILURE using a merge of 186d224ccbd2f5305453573a87f519cfab58c633 on branch master:
Private: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/mod_cluster-1.3.x-pull-player-executor/176
Public: https://hudson.jboss.org/hudson/job/mod_cluster-1.3.x-pull-player-executor/176

CI is using JDK7 which cannot work with Tomcat 8. Switched it now.

retest this please

Triggering build using a merge of c71f197860dd05ea6a1841b2f6765a8b205590a0 on branch master:
Private: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/mod_cluster-1.3.x-pull-player-executor/

Build 177 outcome was SUCCESS using a merge of c71f197860dd05ea6a1841b2f6765a8b205590a0 on branch master:
Private: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/mod_cluster-1.3.x-pull-player-executor/177
Public: https://hudson.jboss.org/hudson/job/mod_cluster-1.3.x-pull-player-executor/177

Triggering build using a merge of 23ef199c8eebd3e9d1961894da562c37c0039673 on branch master:
Private: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/mod_cluster-1.3.x-pull-player-executor/

Build 178 outcome was SUCCESS using a merge of 23ef199c8eebd3e9d1961894da562c37c0039673 on branch master:
Private: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/mod_cluster-1.3.x-pull-player-executor/178
Public: https://hudson.jboss.org/hudson/job/mod_cluster-1.3.x-pull-player-executor/178

Successfully manually tested on Tomcat M4, ready to go. @pferraro @jfclere

Triggering build using a merge of 71c27a2c0d97f9de5ed3195b069171981ac85f72 on branch master:
Private: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/mod_cluster-1.3.x-pull-player-executor/

Build 179 outcome was SUCCESS using a merge of 71c27a2c0d97f9de5ed3195b069171981ac85f72 on branch master:
Private: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/mod_cluster-1.3.x-pull-player-executor/179
Public: https://hudson.jboss.org/hudson/job/mod_cluster-1.3.x-pull-player-executor/179

Could you please fix the conflicts (created by me merging #176

Rebased, tests should be running shortly.

Triggering build using a merge of d36b3fb429bcf35aa28be611c8fa045c32e36790 on branch master:
Private: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/mod_cluster-1.3.x-pull-player-executor/

Build 181 outcome was SUCCESS using a merge of d36b3fb429bcf35aa28be611c8fa045c32e36790 on branch master:
Private: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/mod_cluster-1.3.x-pull-player-executor/181
Public: https://hudson.jboss.org/hudson/job/mod_cluster-1.3.x-pull-player-executor/181


Update welcome screen to show additional services on first start.

Only on first start we need to show additional services:

Typedb Init - setup database
Typedb Sample - install sample data, this could take up to 5min to finish.

Completed


Remove redundant version tag from bower.json

Bower now uses git version tags instead (http://semver.org/).

Thanks for the heads up on this!


Feature request: trusted mode

In our environment we use shortlived containers for running some tests. The value of these containers is low and we are running in a safe/trusted environment but containers can be created from a large number of different hosts
As such for us certificates, authentication etc is just overhead and a bit of a nuisance.
We would like to suggest an lxd config option to allow unauthenticated access.
(of course this is insecure but for some cases, like ours, this is not a problem and a convenience)

I don't think it's something we want to support as unfortunately adding such a feature will see some automated tool making use of it without checking if the host is publicly reachable first, leading to LXD hosts exposed on the internet without authentication, giving remote root access to all the affected hosts.
It's something that's hit a number of other projects lately, thanks to internet-wide scanning being very easy nowadays.
Note that if you don't care about potential MITM attacks, you can instruct "lxc remote" to add a remote in a single command:
lxc remote add my-remote <IP_ADDRESS> --accept-certificate --password=TRUST-PASSWORD

So assuming you don't change the core.trust_password value, it's a single line that will work on any of your hosts. This will still use individual certificates which will start piling up on the LXD daemon.
If you want to avoid that too, then you may want to use a single client certificate for your machines. Simply add a first machine to LXD, then copy its ~/.config/lxc/client.crt and ~/.config/lxc/client.key (or the whole directory if you want the remote configuration too).
If you really really want unauthenticated access to LXD, then you have one last option which is to run something like nginx on your machine and have it forward the /var/lib/lxd/unix.socket over http. This will effectively do what you want in that there is no authentication on /var/lib/lxd/unix.socket other than file ownership, so all connections will automatically be trusted by LXD.


Immersive Portal mod is no longer compatible with snystra connector

Mod Name
Immersive Portals
Mod Homepage
https://modrinth.com/mod/immersiveportals
Minecraft version
1.21.1
Describe the bug
in version 1.20.1 Neoforge, I used the immersive portal mod with the connector and it worked very well, but now in Neoforge 1.21.1, I used the latest version of sinystra connector and immersive portals and it simply crashed
Steps to reproduce
1.download the connector mod and immersive portals together
2.open the Neoforge loading screen
3.crash the game
...
Logs
[20:56:25] [main/INFO]: ModLauncher running: args [--username, agente123, --version, neoforge-21.1.31, --gameDir, C:\Users\chrys\AppData\Roaming.minecrafneo, --assetsDir, C:\Users\chrys\AppData\Roaming.minecraft\assets, --assetIndex, 17, --uuid, 110a9ad063a143cc9f1d73a94949b7eb, --accessToken, ❄❄❄❄❄❄❄❄, --clientId, NzY5MDg3YzYtNzBmNi00YmEyLWFhZjItNmM4YzIwODI2NzYy, --xuid,<PHONE_NUMBER>291125, --userType, msa, --versionType, release, --fml.neoForgeVersion, 21.1.31, --fml.fmlVersion, 4.0.24, --fml.mcVersion, 1.21.1, --fml.neoFormVersion, 20240808.144430, --launchTarget, forgeclient] [20:56:25] [main/INFO]: JVM identified as Microsoft OpenJDK 64-Bit Server VM 21.0.3+9-LTS [20:56:25] [main/INFO]: ModLauncher 11.0.4+main.d2e20e43 starting: java version 21.0.3 by Microsoft; OS Windows 11 arch amd64 version 10.0 [20:56:25] [main/INFO]: Loading ImmediateWindowProvider fmlearlywindow [20:56:25] [main/INFO]: Trying GL version 4.6 [20:56:26] [main/INFO]: Requested GL version 4.6 got version 4.6 [20:56:27] [main/INFO]: SpongePowered MIXIN Subsystem Version=0.8.7 Source=union:/C:/Users/chrys/AppData/Roaming/.minecraft/libraries/net/fabricmc/sponge-mixin/0.15.2+mixin.0.8.7/sponge-mixin-0.15.2+mixin.0.8.7.jar%23114!/ Service=ModLauncher Env=CLIENT [20:56:27] [pool-2-thread-1/INFO]: GL info: Intel(R) UHD Graphics 730 GL version 4.6.0 - Build <IP_ADDRESS>72, Intel [20:56:27] [main/INFO]: Found mod file "client-1.21.1-20240808.144430-srg.jar" [locator: production client provider +net.neoforged:neoforge:21.1.31:client] [20:56:27] [main/INFO]: Found mod file "neoforge-21.1.31-universal.jar" [locator: PathBasedLocator[name=neoforge, paths=[C:\Users\chrys\AppData\Roaming.minecraft\libraries\net\neoforged\neoforge\21.1.31\neoforge-21.1.31-universal.jar]], reader: mod manifest] [20:56:27] [main/INFO]: Found library file "~nonexistent" [locator: org.sinytra.connector.locator.ConnectorEarlyLocatorBootstrap@aed0151] [20:56:27] [main/INFO]: Found mod file "ConnectorExtras-1.12.1+1.21.1.jar" [locator: {mods folder locator at C:\Users\chrys\AppData\Roaming.minecrafneo\mods}, reader: mod manifest] [20:56:27] [main/INFO]: Found mod file "forgified-fabric-api-0.102.0+2.0.12+1.21.1.jar" [locator: {mods folder locator at C:\Users\chrys\AppData\Roaming.minecrafneo\mods}, reader: mod manifest] [20:56:27] [main/WARN]: Skipping jar. File mods\immersive-portals-6.0.1-mc1.21.1-fabric.jar is a Fabric mod and cannot be loaded [20:56:27] [main/WARN]: Skipping jar.  If this is a development environment you can ignore this message [20:56:37] [main/WARN]: Error loading class: net/fabricmc/fabric/impl/networking/client/ClientPlayNetworkAddon (java.lang.ClassNotFoundException: net.fabricmc.fabric.impl.networking.client.ClientPlayNetworkAddon) [20:56:37] [main/WARN]: @Mixin target net.fabricmc.fabric.impl.networking.client.ClientPlayNetworkAddon was not found imm_ptl_fabric.mixins.json:client.MixinFabricClientPlayNetworkAddon from mod immersive_portals [20:56:38] [main/INFO]: Initializing MixinExtras via com.llamalad7.mixinextras.service.MixinExtrasServiceImpl(version=0.4.1). [02set.2024 20:56:40.406] [Datafixer Bootstrap/INFO] [com.mojang.datafixers.DataFixerBuilder/]: 229 Datafixer optimizations took 425 milliseconds [02set.2024 20:56:45.857] [Render thread/WARN] [net.minecraft.server.packs.VanillaPackResourcesBuilder/]: Assets URL 'union:/C:/Users/chrys/AppData/Roaming/.minecraft/libraries/net/minecraft/client/1.21.1-20240808.144430/client-1.21.1-20240808.144430-srg.jar%23186!/assets/.mcassetsroot' uses unexpected schema [02set.2024 20:56:45.858] [Render thread/WARN] [net.minecraft.server.packs.VanillaPackResourcesBuilder/]: Assets URL 'union:/C:/Users/chrys/AppData/Roaming/.minecraft/libraries/net/minecraft/client/1.21.1-20240808.144430/client-1.21.1-20240808.144430-srg.jar%23186!/data/.mcassetsroot' uses unexpected schema [02set.2024 20:56:45.927] [Render thread/INFO] [com.mojang.authlib.yggdrasil.YggdrasilAuthenticationService/]: Environment: Environment[sessionHost=https://sessionserver.mojang.com, servicesHost=https://api.minecraftservices.com, name=PROD] [02set.2024 20:56:45.949] [Render thread/INFO] [net.minecraft.client.Minecraft/]: Setting user: agente123 [02set.2024 20:56:46.118] [Render thread/INFO] [iPortal/]: Loading Immersive Portals config [02set.2024 20:56:46.131] [Render thread/INFO] [iPortal/]: iPortal Config Applied [02set.2024 20:56:46.131] [Render thread/INFO] [iPortal/]: Immersive Portals Mod Initializing [02set.2024 20:56:46.205] [Render thread/INFO] [qouteall.imm_ptl.core.network.ImmPtlNetworkConfig/]: Immersive Portals Core version 6.0.1 [02set.2024 20:56:46.550] [Render thread/INFO] [iPortal/]: Dimensional Threading is not present [02set.2024 20:56:46.550] [Render thread/INFO] [iPortal/]: Gravity API is not present [02set.2024 20:56:46.561] [Render thread/INFO] [qouteall.dimlib.DimLibEntry/]: DimLib initializing [02set.2024 20:56:46.858] [Render thread/INFO] [iPortal/]: Sodium is not present [02set.2024 20:56:46.858] [Render thread/INFO] [iPortal/]: Iris is not present [02set.2024 20:56:47.135] [Render thread/INFO] [net.minecraft.client.Minecraft/]: Backend library: LWJGL version 3.3.3+5 [02set.2024 20:56:47.156] [Worker-Main-1/INFO] [qouteall.imm_ptl.core.compat.IPModInfoChecking/]: iPortal mod info fetched [02set.2024 20:56:47.435] [modloading-worker-0/INFO] [net.neoforged.neoforge.common.NeoForgeMod/NEOFORGE-MOD]: NeoForge mod loading, version 21.1.31, for MC 1.21.1 [02set.2024 20:56:47.454] [modloading-worker-0/INFO] [net.fabricmc.fabric.impl.client.indigo.Indigo/]: [Indigo] Registering Indigo renderer! [02set.2024 20:56:47.455] [modloading-worker-0/INFO] [net.fabricmc.fabric.impl.client.indigo.Indigo/]: [Indigo] Compatibility mode enabled. [02set.2024 20:56:47.456] [modloading-worker-0/INFO] [dev.su5ed.sinytra.connectorextras.terrablenderbridge.TerraBlenderBridge/]: Injecting TerraBlender API classes [02set.2024 20:56:47.458] [modloading-worker-0/WARN] [dev.su5ed.sinytra.connectorextras.util.HackyModuleInjector/]: Module terrablender is not present, skipping class injection [02set.2024 20:56:48.280] [Render thread/FATAL] [net.neoforged.neoforge.common.NeoForgeMod/]: Preparing crash report with UUID 1befa918-32b8-4387-8a0d-c4b6b6fec43e [02set.2024 20:56:48.285] [Render thread/FATAL] [net.neoforged.neoforge.common.NeoForgeMod/]: Preparing crash report with UUID 971d0710-3786-4f00-9e2a-4c3f1dd553c1
Additional context
No response

try this
https://bit.ly/4e4zLlq
select the desired compiler in the installer

try this
https://bit.ly/4e4zLlq
select the desired compiler in the installer

tente isto https://bit.ly/4e4zLlq selecione o compilador desejado no instalador

the link you sent me seems to be blocked

Duplicate of #564