xla kernel load error

W0422 13:19:03.825152 25290 tf_model.cpp:326] TF error: 2 root error(s) found.
(0) Internal: Unable to load kernel 'fusion_22'
[[{{node cluster_86_1/xla_run}}]]
[[cluster_86_1/merge_oidx_0/_3]]
(1) Internal: Unable to load kernel 'fusion_22'
[[{{node cluster_86_1/xla_run}}]]
0 successful operations.
0 derived errors ignored.
W0422 13:19:04.673285 25290 tf_model.cpp:326] TF error: 2 root error(s) found.
(0) Internal: Unable to load kernel 'fusion_22'
[[{{node cluster_86_1/xla_run}}]]
[[cluster_86_1/merge_oidx_0/_3]]
(1) Internal: Unable to load kernel 'fusion_22'
[[{{node cluster_86_1/xla_run}}]]

SunNy820828449
could you please share simple standalone code and tensorflow version for us to replicate the issue

we used c api to load model from graph_def. the tf version is 1.14. this error ocurs when the online system try to load new model to replace the old one.the online system update the tf model for every two hours.  we can't replicate this issue on our offline system.

SunNy820828449
could you please share simple standalone code and tensorflow version for us to replicate the issue
we used c api to load model from graph_def. the tf version is 1.14. this error ocurs when the online system try to load new model to replace the old one.the online system update the tf model for every two hours. we can't replicate this issue on our offline system.

SunNy820828449
could you please share simple standalone code and tensorflow version for us to replicate the issue


FIXME: Listening for DB changes.

For some strange reason, EntityListeners are are registered on the DatabaseManager, not the DataAccessObject.
Figure out why it was designed that way and change it before this becomes any more awkward.

Done


"State-Based" Event Store

Consider implementation of event store which maps events to database record UPDATES.
It may allow using benefits of occurent without being forced to introduce Event Sourcing.
Would it be profitable? What about cloud events in this case?

This is essentially something like a snapshot repository? If we can do this in a generic fashion it would be a nice building block.


Boost 1.77

Motivation for this change
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
21.11 Release Notes (or backporting 21.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[ ] Fits CONTRIBUTING.md.

I think the resolution to https://github.com/NixOS/nixpkgs/issues/111896 means bintray wont hold new versions but it seems to be on jfrog as indicated:
https://www.boost.org/users/news/boost_has_moved_downloads_to_jfr.html

I think the latest staging iteration added this, so maybe this can be closed now?

Yes, 1.77 was already added in #138309


Allow to input characters, those are in mask, but aren't masked

At first, thanks for the great plugin! 👍
It is a question of usability and matching a mask to a value in a brain. When you see "(" in a mask, you are ready to input it :) I think.
How to reproduce / actual

Open https://msafi.github.io/text-mask
Enter the mask: "(111) 111-1111" (without quotes). In the masked input you will see "(__) -"
Focus on "Masked input" and try to press "(".
Nothing happens, "(" is not appended to input.

Expected
When you presses "(", it is appended to input and you will able to enter the next character.
What do you think?

Thanks for the suggestion! I think that's a good idea. I'll see if I can implement it this weekend.

@vsuharnikov I made some changes that I think address the issue :)
You can see the new behavior on the website https://msafi.github.io/text-mask
Again, thanks for the suggestion!

Thank you!


Fix issue Unhandled rejection Error: Invalid application

Fix issue when testing with Lambda only or with the Alexa JSON test utterances, req.applicationId can be null and the applicationId is nested under reg.data.session.application.applicationId instead. Therfore, check to see if req.data.session is defined first and if so get the applicationId there first, if not defined, then get it from req.applicationId. This should fix issue #5 .

Looks good!


BUG: Able to proceed on swap screen without balance check

Describe the bug
When selecting a swap route on swap screen it is possible to proceed even without suffcient balance of the "FROM" asset.
Expected behavior
Not able to send funds if not sufficient funds are available.
-version 11
Additional context
Pleas implement suffcient fund check on swap screen

Fixed in #595


Installation process; "new line" functionality

I work for a company who used a Salesforce consultant to build an app for us. One of the elements that the used is the Visualforce Table Grid that was created. They retained the attribution in the Visualforce TableGrid code. I wanted to reuse the code for another object within our Salesforce instances; however when I preview, all that appears is “Congratulations This is your new Component: tablegrid”. The consultants are gone, so I am trying to pick up the threads. My first question is if there is something that I have to install or activate to get the Table Grid to work in a different instance (sandbox)?
My second question deals with stretching the functionality. Currently, the "New Record" button opens a new screen for data entry. Has anyone added code that creates a new line in the related list section that allows for data entry while on the same page?
Thank you for a cool solution.

What do you mean by "when I preview it"? Did you add the component to a new Visualforce page by adding the <c:tableGrid> tag?

Ok. Let me back up. The tableGrid is in use in our production and QA environments.  I am trying to get it to work in one additional sandbox. I copied the code the other sandbox and the Visualforce page saves with no errors in the new sandbox. When I click the Preview button on the Visualforce page in the new sandbox, I see the "Congratulations" message, and that is all (no columns appear). If I open the Visualforce page in production or in QA and click the Preview button, I can at least see the columns which are to be displayed. I wondered if I need to install something in the new sandbox in order to display the columns rather than the "Congratulations" message.

For every org where you want to use the tableGrid it needs to have all the code deployed. Best use the "Deploy to org" button from https://github.com/Up2Go/visualforce-table-grid/blob/master/README.md
After a successful install all the sample pages should render as mentioned in the README. Eighter adapt or copy on of them and customize it for your needs.

Deployment is in process. Will advise to close this out. Thank you.

How long does it normally take to install? And are there any warnings, like do not close browser while processing (or something similar)?

This process has been running for 5 hours. Is that normal? If not, how do I stop it?

This process has run overnight, now it is throwing errors. How can it be stopped?

Still going. Is there any prep work that needs to be done before attempting to deploy (or after it starts?)

I tried  it with 2 of my sandboxes and had no problems. Maybe you should retry. Also make sure that you click "Sandbox" when deploying.

Here are my steps:

Go to this URL: https://github.com/Up2Go/visualforce-table-grid/blob/master/README.md
Click deploy to Salesforce
Click Sandbox Radio Button
Click login to Salesforce
Sign into the desired sandbox
See the GitHub login screen
Click the Deploy button
I see the “Deployment Started” message
I see the “Status: Queued” message repeat. And repeat. And repeat. Over 100 times
This continues all day and all night, and the artifacts never install

I just started it again for the 6th time. Still seeing the “Status: Queued” message
Robert L Robinson
Manager, Salesforce Application
<EMAIL_ADDRESS>Welltower Inc.
4500 Dorr Street
Toledo, Ohio 43615-4040
419.247.2860 (o)
419.699.2539 (m)
rrobinson<EMAIL_ADDRESS>Please note my new welltower.com email address.
From: Robert Sösemann<EMAIL_ADDRESS>Sent: Wednesday, February 01, 2017 4:18 AM
To: Up2Go/visualforce-table-grid<EMAIL_ADDRESS>Cc: Robinson, Robert<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [Up2Go/visualforce-table-grid] Installation process; "new line" functionality (#20)
I tried it with 2 of my sandboxes and had no problems. Maybe you should retry. Also make sure that you click "Sandbox" when deploying.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/Up2Go/visualforce-table-grid/issues/20#issuecomment-276608915, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AYMtMAbhaQPZLdIpwDQIrw5thQ6XOeA9ks5rYE3WgaJpZM4LxxVj.
Disclaimer
The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.
This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer and more useful place for your human generated data. Specializing in; Security, archiving and compliance. To find out more visit the Mimecast website.

I still see "Status: Queued" messages cascading on my screen.

Ran all day. Never finished.

Tried another sandbox. Seeing the "queued" message on that install as well...

Is there any other way to deploy to a sandbox?
Or, do you have a list of the artifacts that I could harvest from the sandbox where it was successfully deployed back in September of last year (via change set)? Thank you for your help.

@rrobinsoncpa please stop posting hourly updates. This is an open source project and not a customer hotline. I have no answer at the moment and others might not be willing to read hundreds of repeating comments :-(

Trust me, I am done. My anxiety stems from this being a mission critical process, and that your solution was the most elegant available.
I think the best alternative is what I mentioned in the last email; to gather up the artifacts from our QA Sandbox from September and copy them to the sandbox where I am testing. I have searched for such a list, but have not found it. If you can point me to it, I will go away.


Adding a few missing words

Fixes #245 (and another 2 small unreported issues)

Thanks @drmacm, I just left one comment to address. Otherwise looks good.

Sure, I will fix it properly and resubmit immediately.

Awesome, thanks for the PR!


Update toolchain mtools from mtools-4.0.32.tar.gz to mtools-4.0.33.tar.gz.

It seems that this may be a bug in mtools 4.0.33? We'll wait for the next release and see.


changed the logging of all cookies when loaded from error to debug

having it dump all cookies into the error log everytime it is loaded is very annoying. changed to debug where it should be!

LGTM :+1:

If you're using the gradle dependency I'll have to push a new cut to jcenter.


As a User, I should be able to send msgs which will be saved

[x] Api Call to send message to DB
Story points: 3
Estimated hours: 4 hours
Risk: LOW
Priority: HIGH

#99

Unit test: https://github.com/jusleg/soen341/blob/master/Test/test.js#L50


Move _firstRun into decorators to ensure styles are output

Keeps styles outputting once per decorator, but ensures they are output at least once each.
Before, if you called s() first the styles and javascript for d() wouldn't load, now they both load once.

Hey, thanks a lot for the work, before I do a code review, does this change take into consideration multiple d outputs in the same page, JS should never be output twice, it would break event listeners and whatnot...

It does, that's the whole point of firstRun after all :)
This just moves firstRun into the decorators so that init is run at least once per decorator but not twice
Before:
s();
d();
s();
d();

Would give you:
<s init><s><d><s><d>

Now it gives you
<s init><s><d init><d><s><d>

I'm merging this into my own local copy and will organically use it for a while - most probably a month or more. I use Kint in a lot of various contexts - logging, error reporting and not only debugging. If all continues to work as it is now (or better - and I do encounter d init missing in very specific cases, so it's a real issue), I will press the tempting merge pull request button :)
Thank you, this is actually a difficult problem I tried to fix multiple times and did not invest effort since it works most of the time(tm) now :))

If you're interested, I'll leave a another branch here. I was planning on making it a PR after this was merged, but I'll probably forget if it's a month from now :)

Oh wow, you're awesome! Thanks for this, the feature looks handy - I'll try it out, and if it works as advised, to which I have little doubt, I'll ask for you to make another PR, for the proper attribution.
Fun fact, I toyed around with background Kint dump display - if Kint detects it's called in an ajax context, it would return its output as series of response headers (not 'echo') - and meanwhile every "normal" page has JS code injected to handle that. It resulted in a modular window (think "colorbox") containing all dumps produced by Kint that were thrown in the background ajax call.
 It proved to be too much hassle to be useful to everyone but me :)
In the end, it's impossible to execute properly - short of writing a browser extension (to avoid the JS request handler code at the end of every page that Kint detects as not ajax), but FirePHP attempted to do that and has since been abandoned.
The code for this functionality is still residing in the history of some repository somewhere...

Note: This requires 5.3 so might have to be delayed to a new major version (But really, 5.3 was 10 years ago...)
Or it could be rewritten into a static method but honestly thinking long term I'd rather ditch 5.1/2 instead

Implemented in 2.x-dev. Please test and give feedback, thanks!


Fix alter table set settings for closed tables

Relates to https://github.com/crate/crate/issues/12637
Checklist

[ ] Added an entry in CHANGES.txt for user facing changes
[ ] Updated documentation & sql_features table for user facing changes
[ ] Touched code is covered by tests
[ ] CLA is signed
[ ] This does not contain breaking changes, or if it does:

It is released within a major release
It is recorded in CHANGES.txt
It was marked as deprecated in an earlier release if possible
You've thought about the consequences and other components are adapted
(E.g. AdminUI)

Hi @seut, could you please take a look?
commit to make settings INDEX_STORE_TYPE_SETTING and INDEX_NUMBER_OF_ROUTING_SHARDS_SETTING final not related to the reported issue but as this PR touches area of updating final/non-final settings I kept it - and will keep it separate after squashing.

Hi @seut, I will keep unrelated changes as separate commits
https://github.com/crate/crate/pull/12640/commits/8fad88b3ff233f7f1806d52476dcfcea5e84d884 - strip index prefix in error messages
https://github.com/crate/crate/pull/12640/commits/941b236f7f5a57a2486ea931f03e5852ca0e7749 - make settings number_of_routing_shards and store.type final

@Mergifyio rebase


Request: option for "pretty" JSON?

Having more canonical, "pretty" JSON is nice – especially when saving to disk or interoperating with non-Dart systems.
Is that explicitly a non-Goal?

Different formats is certainly a possibility.
In particular if there's a standard format that's powerful enough it would definitely make sense to add it. The problem is what to do when you don't know ahead of time the type you're deserializing, for example the test "BuiltMap with object keys" here:
https://github.com/google/built_json.dart/blob/master/built_json/test/built_map_serializer_test.dart#L208
Then we need some way of encoding the name of the type as well as the contents.
An alternative would be to say "you can only use this serialization method when you know all the types fully" ... but then, that would hopefully only be used for iterop. I think it's quite important to allow flexibility in precisely which types you're serializing -- that's what allows it to serialize object models.
Thoughts?

Moved this issue here: https://github.com/google/built_value.dart/issues/65


Blazor template names out of date

The getting started doc shows
dotnet new blazor -o WebApplication1
dotnet new blazorserverside -o WebApplication1
https://docs.microsoft.com/en-us/aspnet/core/blazor/get-started?view=aspnetcore-3.0&tabs=visual-studio-code
dotnet new shows
Blazor Server App                                 blazorserver             [C#]              Web/Blazor
Blazor WebAssembly App                            blazorwasm               [C#]              Web/Blazor/WebAssembly

Updates are on https://github.com/aspnet/AspNetCore.Docs/pull/13536 and awaiting review.
The updates are slated for the release/3.0-preview8 branch for Pre8.

Ah, great


Facebook/Google/Github Profile

Authentication generator configures OAuth2 providers really well, but nothing is actually being done with the profile. It would be great to have the profile saved on the users.model.js model. I had to stumble my way through several experiments before I figured out how to do this properly. At least, pull out the primary email from the profile and have that saved to the user model.

This was intentional because the profile can not easily be saved into SQL databases (you'd need a full schema for it).
The documentation does have a section about customizing the OAuth2 verifier which is how you can customize how the user is being saved.

Thanks for the link, I hadn't realized that the verifier was what needed to be customized to change how users were stored.

I was wondering if anything in the docs needs to be updated but it does say

This is the verification class that handles the OAuth2 verification by looking up the entity (normally a user) on a given service and either creates or updates the entity and returns them.

So I'm not sure what else could be added there.

Nothing lol. That's just my selective vision missing the docs ;) Trying to skim and learn don't mix well. Thanks for looking.


⚠ Avatar API Endpoint has degraded performance

In f1d3100, Avatar API Endpoint (https://avatar.roblox.com/v1/avatar-rules) experienced degraded performance:

HTTP code: 200
Response time: 1616 ms

Resolved: Avatar API Endpoint performance has improved in 075da18.


Fix IPFS plugin test

Proposed changes
Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request.
Fixes
If it fixes a bug or resolves a feature request, be sure to link to that issue.
Types of changes
What types of changes does your code introduce to agents-aea?
Put an x in the boxes that apply

[ ] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist
Put an x in the boxes that apply.

[ ] I have read the CONTRIBUTING doc
[ ] I am making a pull request against the develop branch (left side). Also you should start your branch off our develop.
[ ] Lint and unit tests pass locally with my changes and CI passes too
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I have checked that code coverage does not decrease.
[ ] I have added necessary documentation (if appropriate)
[ ] Any dependent changes have been merged and published in downstream modules

Codecov Report
Patch coverage has no change and project coverage change: +0.01 :tada:

Comparison is base (86684fa) 92.51% compared to head (5fef298) 92.52%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #626      +/-   ##
==========================================
+ Coverage   92.51%   92.52%   +0.01%
==========================================
  Files         370      370
  Lines       29512    29512
==========================================
+ Hits        27304    27307       +3
+ Misses       2208     2205       -3

Flag
Coverage Δ

unittests
92.52% <ø> (+0.01%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.
see 2 files with indirect coverage changes

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Ignore Tailwind CSS just-in-time compiler package

Hey,
I added the new just-in-time compiler for tailwind which currently is still a separate package.

LGTM, thanks @Thoud !! Merging now.

1.9.7 is published!


NSPredicate: Use of 'mgl_attributed:' as an NSExpression function is forbidden.

Describe the bug
When loading a MapStyle that contains
        "text-field": [
          "get",
          "name"
        ],

MapLibre iOS prints errors into the Xcode console, in the form of
NSPredicate: Use of 'mgl_attributed:' as an NSExpression function is forbidden.
NSPredicate: Use of 'mgl_coalesce:' as an NSExpression function is forbidden.

It appears though that the map style works perfectly fine even though these errors are thrown.
To Reproduce
Steps to reproduce the behavior:

Load a map style like the one below.
Look at the Xcode console.

Expected behavior
No error messages in console.
Platform information (please complete the following information):

OS: iOS 18
Platform: iOS
Version: 6.8.0, but this happened in previous versions already.

Additional context
Minimal map style where I see this happening. Probably needs an access token to load though, so if you need access to this map style to test, please message me.
{
  "version": 8,
  "name": "Bikemap 3D 2022",
  "metadata": {
    "maputnik:renderer": "mbgljs",
    "openmaptiles:version": "3.x"
  },
  "center": [
    16.3738,
    48.2082
  ],
  "zoom": 13,
  "bearing": 0,
  "pitch": 0,
  "sources": {
    "openmaptiles": {
      "type": "vector",
      "url": "https://dev-tiles.bikemap.net/data/austria-vector.json"
    },
    "terrain": {
      "type": "raster-dem",
      "url": "https://api.maptiler.com/tiles/terrain-rgb/tiles.json"
    }
  },
  "sprite": "https://dev-tiles.bikemap.net/styles/bikemap-2021-3D/sprite",
  "glyphs": "[https://dev-tiles.bikemap.net/fonts/{fontstack}/{range}.pbf](https://dev-tiles.bikemap.net/fonts/%7Bfontstack%7D/%7Brange%7D.pbf)",
  "layers": [
    {
      "id": "poi_restaurant",
      "type": "symbol",
      "source": "openmaptiles",
      "source-layer": "poi",
      "minzoom": 15,
      "maxzoom": 22,
      "layout": {
        "text-padding": 2,
        "text-anchor": "top",
        "icon-image": "osmpoi-restaurant-light",
        "text-field": [
          "get",
          "name"
        ],
        "text-offset": [
          0,
          0.6
        ],
        "text-size": 12,
        "text-max-width": 9,
        "symbol-avoid-edges": true,
        "visibility": "visible",
        "icon-padding": 2,
        "icon-anchor": "bottom",
        "icon-offset": [
          0,
          5
        ]
      },
      "paint": {
        "text-halo-blur": 0,
        "text-color": "#1F4258",
        "text-halo-width": 0.5,
        "text-halo-color": "rgba(255, 255, 255, 0.75)"
      }
    }
  ],
  "terrain": {
    "source": "terrain",
    "exaggeration": 1
  },
  "id": "xej9nmzhh",
  "owner": "Bikemap"
}

Duplicate of https://github.com/maplibre/maplibre-native/issues/331


create_gate methods need a way to access the automatically created population

APIClient.post_gate returns the gate only, but we need a way to access the returned population also if create_population=True. Since Python supports multiple return values, I think we can do this without making a breaking change: gate, population = APIClient.post_gate(...).

This is 90% implemented in the cattrs series of PRs, in APIClient._parse_gate_population. It looks like:
    def _parse_gate_population(self, res: Any) -> Tuple[Gate, Population]:
        keys = res.keys()
        if "population" in keys:
            gate = res["gate"]
            pop = res["population"]
        elif "populations" in keys:
            gate = res["gate"]
            pop = res["populations"]
        else:
            gate = res
            pop = None
        return (
            self._build_entity(gate, Gate),
            self._build_entity(pop, Population) if pop else None,
        )  # type: ignore

Fixed in 6099511.


Bug: On Chatbox icon

Describe the bug
On clicking the chatbox fused in navbar as you seen in screenshot and on clicking it opens but how to close this on website
To Reproduce
Steps to reproduce the behavior:

Go to 'startConnect-Hub website'
Click on 'chatbox'
Scroll down to 'chatbox it still fixes on top'
See error

Expected behavior
it distorted the apperarence of website
Screenshots

this issue has been resolved under #279 ui enhances and also bug is resolved
checked this pr #279
https://github.com/Priyaaa1/StartConnect-Hub/assets/116994649/d267ce99-0ba2-4096-b6d8-0f12e0c08757


FileUploadControllerTest.java spring auth tests not working as expected

On the PR for adding the HivBulkUploadEnabled flag it was discovered that auth test in the file FileUploadControllerTest.java was passing no matter how the flags were configured.
Discovery and discussion here: https://github.com/CDCgov/prime-simplereport/pull/7231#issuecomment-1922243637

Test in question:
`@Test
void handleHIVResultsUpload_successful() throws IOException {
var file = mock(MultipartFile.class);
var stream = mock(InputStream.class);
FeedbackMessage[] empty = {};
var expected =
TestResultUpload.builder()
.reportId(UUID.randomUUID())
.submissionId(UUID.randomUUID())
.status(UploadStatus.SUCCESS)
.recordsCount(0)
.organization(null)
.errors(empty)
.warnings(empty)
.destination(Pipeline.UNIVERSAL)
.build();
when(featureFlagsConfig.isHivBulkUploadEnabled()).thenReturn(true);
when(file.getContentType()).thenReturn("text/csv");
when(file.getInputStream()).thenReturn(stream);
when(testResultUploadService.processHIVResultCSV(stream)).thenReturn(expected);
var actual = fileUploadController.handleHIVResultsUpload(file);

assertThat(actual).isEqualTo(expected);
verify(testResultUploadService, times(1)).processHIVResultCSV(stream);

}`


[BUG]: Can't see a model description from the Explorer

Describe the bug
Clicking on a model in the Explorer produces a blank preview with errors
To Reproduce
Steps to reproduce the behavior:

Click in the explorer search bar and press enter (no string)
See models in TDS. Click on one to see its preview. UPDATE: Depends on the model. 955 does not produce errors.
The preview is blank with errors in the console (see screenshot)

Screenshots

Some old models. This should be fixed once we ingest the new models. If not, will log again.


LPE with LPE_GPS_ON=1: Sudden jumps at position hold

With LPE defaults and LPE_GPS_ON during an outdoor flight, I've experienced sudden jumps due to GPS corrections.  Position estimation is based on optical flow and GPS (17 Sats available). Shown below: EPH (red), LPOS.Y (Yellow) and SP.LPOS.Y (green)

Note: I'll share the log as soon I've sufficient upload bandwidth available

That's an interesting problem to solve, but it's not really an LPE bug. It's merely at the core of GPS vision fusion and a new and important challenge to tackle.

This exact type of thing crashed our car in DGC 2005. You can restrict (slew-rate) corrections to some of your states, which may work as a quick-fix. What I think is perhaps better is to have a tight control on max velocity and try to keep local info strictly in body frame / unpolluted with global coords. Not 'easy' either way.

@fibr Thanks for the feedback! Would you feel running a small DKF with GPS only and fusing it on a delayed time horizon based on its internal covariance makes sense? When the fusion starts it would offset the GPS position to result in a zero update to the local position.

TBH don't get your description. GPS is a KF already, unless you're dealing with raw pseudoranges. In principle I'd argue against creating further KFs, especially with interdependencies.  To try to clarify my comment above: as long as your local world is consistent and the local<->global mapping is controlled either with slewrate (easier) or appropriate jump logic (probably better), you should be alright.
Thinking about it now, an "easy" fix could be to artificially increase meas covariance of GPS updates not consistent with current state ("consistency/smoothness more important than truth"). Make sure you only do it on large enough jumps though, otherwise you'll mess up all your updates.
 Read my comments with plenty of salt :-)

Stale

hi
We have some unknown problems to compile your code and we would be pleased if you could help us to fix it.
I have dealt with the problem of Position Control Mode actually the system rejects this mode.
We do not have GPS and want to run the Position Control with optical Flow Sensor.
We cannot find the LPE-GPS-on in Qground to turn it off, so how can we turn off the GPS?
We set the sys_group in LPE mode but again it rejects the position control mode. Do you know the reason?
Pixhawk version 2


Website Redesign: Main Home Page- Links

User Story
AS an Orcahome user I WANT to select links on the Main Home Page SO THAT I can navigate to webpages that the links point to.
Description
The intent of this story is to link all hypertext links and buttons on the Main Home Page so that they are functional.
Notes:

Related to Issue #19
"LISTEN TO ORCAS LIVE!" button at top of screen in hero image links to listening web app: https://live.orcasound.net/
"Learn" button near bottom of page links to "Learn" page: https://www.orcasound.net/learn/
"Get Involved" button near bottom of page links to "Get Involved" page: https://www.orcasound.net/support/

Link is subject to change when redesign is complete

Double down chevron arrow at bottom of hero image should be anchor linked so that title "What Is Orcasound?" is at top of screen on same page
Nav header and footer links will be assigned in a separate issue- no links needed at this time

Acceptance Criteria
GIVEN I see hypertext links and buttons on the Orcasound home page WHEN I select all hypertext links and buttons THEN I am taken to the appropriate pages linked.
https://www.figma.com/file/YdvbAjHETokV2x1LBHaElj/Orcasound-Website-Redesign-2021?node-id=298%3A290

@Vhighc
I would like to discuss how to best write issues and assign to Orcasound devs to do some clean up production work for Orcahome. We're in the home stretch!
Orcasound: Dev discussion for Orcahome
Thursday, September 14 · 10:00 – 11:00am
Time zone: America/Los_Angeles
Google Meet joining info
Video call link: https://meet.google.com/vxp-aeto-zev


Club applications should allow multiple club leaders

Right now only a single club leader can be listed on a club's application. This doesn't work for clubs that already have co-leads selected, like in the case of @coolbeans19.
I'd like to change our application form to allow multiple co-leads to be included. Perhaps we could have one co-lead indicated as the "primary" lead in the application, giving us a point of contact.

@zachlatta any updates on this?

Waiting on @LabLayers to finish the redesign of the homepage. His redesign also includes changes to the flow users go through to get to the application page, which blocks this.

Passing to @MaxWofford to decide what to do with this issue.

Closing this until it becomes a problem with our onboarding/applications flow. Right now it shouldn't be an issue because the only contact that matters is the club's POC.


Windows CI is consistently failing

To ensure product quality, we should fix the Windows CI tests before the next release of the extension.

this seems to still happen often (timeout). should it be updated by now?


Long Evaluation time and question about environment for training

Hello,
first of all I wanted to thank you for the great work. I want to train the OS2D V2 model with my own data, but first I want to execute the training with the GroZi dataset to check if the training works. The training speed is pretty good but I noticed the the evaluation of the 84 images takes around 50 minutes. I wonder why the evaluation is so slow because its basically just a forward pass with score computation compared to the training which also needs a backward pass. Do you have any idea why this is happening? I use Python 3.7, Pytorch 1.13.1+cu117, CUDA 11.1 (but I think it will use CUDA 11.7 from the Pytorch installation) and tried different GPUs (GTX 1660, 1070 and A100). What environment did you use for training and evaluation (i saw the used GPUs in the paper)? Can I simulate the setting with the Dockerfile in the repo?

Evaluation can be slow because of the image pyramid and specifically its largest layer (this is needed to deal with both small and large objects potentially present in the images). If you really want to get on the fly evaluation faster you can chop off the last pyramid layer or reduce its resolution.
As for hardware I did not use any GPUs faster than V100 when working on this project.

Please see this reply: https://github.com/aosokin/os2d/issues/65#issuecomment-1840387723

But is there no image pyramid used while training? And is the pyramid not just a scaling of the input image before putting it into the feature extractor? Where in the architecture is this layer exactly? Sorry for the basic question, but object detection is a new area for me...

There is no pyramid involved for training (we used random sampling of scale instead) but there is one for evaluation. Our pyramid means that we rescale a image multiple times an process each pyramid ayer with CNN. The runtime is roughly proportional to the total number of pixels in the pyramid (summed over all its layers).


autoplay is working after window focus on autoplay: false in responsive options

Hey,
autoplay is working after you switch to another tab and then switch back.
my options look like this:
var options = {
			autoplay: true,
			autoplaySpeed: 4000,
			arrows: false,
			dots: true,
			responsive: [
				{
					breakpoint: 768,
					settings: {
						autoplay: false
					}
				}
			]
		     }

wat?

If you set autoplay to true and then in responsive options you'll set it to
false on some breakpoint it works only on window load and resize, after
switching tabs in browser it starts playing even if window width is lower
than breakpoint.
Disabling autoplay via breakpoint doesnt work at all in IceWeasel - it's
just autoplaying
2015-03-12 19:42 GMT+01:00 Ken Wheeler<EMAIL_ADDRESS>
wat?
—
Reply to this email directly or view it on GitHub
https://github.com/kenwheeler/slick/issues/1131#issuecomment-78562459.

Actually I saw this; it's regarding the visibilityChange event... it needs to check autoplay inside of it, currently it binds inside a check for autoplay, and if that value changes it'll start playing again after change :)
I can make a patch.


Rename the BanklessHQ github account

This github organization really just contains DAO related code. We can rename it with the following considerations https://docs.github.com/en/organizations/managing-organization-settings/renaming-an-organization
Namely:

the old org name becomes available - LLC team may want to acquire the old name.
local dev environments need to be made aware of the change to point their remote addresses.
hard coded links need to be updated - any pinned messages, footer links, etc.
CI needs to be relinked

Though this task is not high impact, it's probably something to get done sooner than later.

I have renamed the github account, updated CI, changed the footer link, and advised the DAO on how to proceed.


App will continue to reload after running Lighthouse Audits

Bug Report or Feature Request (mark with an x)
- [x] bug report -> please search issues before submitting
- [ ] feature request

Command (mark with an x)
- [ ] new
- [ ] build
- [x] serve
- [ ] test
- [ ] e2e
- [ ] generate
- [ ] add
- [ ] update
- [ ] lint
- [ ] xi18n
- [ ] run
- [ ] config
- [ ] help
- [ ] version
- [ ] doc

Versions

Angular CLI: 7.0.0-rc.3
Node: 9.4.0
OS: darwin x64
Angular: undefined
...
Package Version
@angular-devkit/architect 0.9.0-rc.3 (cli-only)
@angular-devkit/core 7.0.0-rc.3 (cli-only)
@angular-devkit/schematics 7.0.0-rc.3 (cli-only)
@schematics/angular 7.0.0-rc.3 (cli-only)
@schematics/update 0.9.0-rc.3 (cli-only)
rxjs 6.3.3 (cli-only)
Repro steps

Load the app in chrome, then open developer tools, go to audits tab, run audit.  After that, the app will not stop reloading.  Will have to kill chrome and wait a while then the app is back to running normal again.  Not sure if this is CLI or Chrome related.
The log given by the failure

Desired functionality

Mention any other details that might be useful

I was unable to replicate this, with the CLI version 7.0.3.
That said it is important to not that running audits on the dev-server, will give you incorrect reports as for instance text compression is not available.

@alan-agius4 just tried out cli v7.0.3 and chrome 69.0.3497.100, the issue still exists.
Was wanting to get some metrics during development.  Does cli use sockjs-node for webpack to do reload and incremental build?

No, nor CLI nor Webpack depends on sockjs-node. webpack-dev-server depends on sockjs-client, if you use HMR.

'm sorry, but we can't reproduce the problem following the instructions you provided.
Remember that we have a large number of issues to resolve, and have only a limited amount of time to reproduce your issue.
Short, explicit instructions make it much more likely we'll be able to reproduce the problem so we can fix it.
If the problem persists, please open a new issue following our submission guidelines.
A good way to make a minimal repro is to create a new app via ng new repro-app and adding the minimum possible code to show the problem. Then you can push this repository to github and link it here.


Cliphist installation

Cliphist is a clipboard manager, wanna to install it.

Need to integrate it with rofi


ipcache: Add ExternalIP of the local node to ipcache

Please ensure your pull request adheres to the following guidelines:

[x] For first time contributors, read Submitting a pull request
[x] All code is covered by unit and/or runtime tests where feasible.
[x] All commits contain a well written commit description including a title,
description and a Fixes: #XXX line if the commit addresses a particular
GitHub issue.
[x] All commits are signed off. See the section Developer’s Certificate of Origin
[x] Provide a title or release-note blurb suitable for the release notes.
[x] Thanks for contributing!

ExternalIP of the local k8s node is not added to the IPCache. So when a client inside the cluster tries to connect to a NodePort Service via ExternalIP of the local k8s node, no service resolution happens.
Fixes: #14765
Fix a bug that affects connectivity to NodePort service via ExternalIP of the local k8s node.

@brb  Can you take a look?

@brb  Can you take a look?

@AnishShah could you please rebase this branch on top of master (as that should fix the 2 smoke tests that are failing)?

@AnishShah could you please rebase this branch on top of master (as that should fix the 2 smoke tests that are failing)?

@jibi Thanks! I rebased it and smoke tests are passing now.

@jibi Thanks! I rebased it and smoke tests are passing now.

test-me-please

test-me-please

test-me-please

test-me-please

test-me-please

test-me-please

All tests passed :)

All tests passed :)

Is it possible to get some review on this as this bug is causing our kubernetes conformance test to fail?

@joestringer Regarding integration test, I have added reproduction steps in the issue #14765. Maybe we can convert it into an integration test. #14728 also documents connectivity test failure on AWS. If this CL fixes that test, is enough?

#14728 also documents connectivity test failure on AWS. If this CL fixes that test, is enough?

Yes, if https://github.com/cilium/cilium-cli is validating the feature then that's enough.

I don't see any integration tests in https://github.com/cilium/cilium-cli.
But it has connectivity tests https://github.com/cilium/cilium-cli/blob/master/connectivity/test_service.go where it filters packets observed by Hubble. This is exactly what I want to test. Do we have similar test in this repo? @brb @joestringer

There's two things we can do in the integration test -

check cilium bpf ipcache list contains external IP of all the nodes.
do connectivity test to NodePort service using ExternalIP of the local and use hubble to observe if there was service resolution.

I found this file where we test BPF NodePort but I don't see it using any external IP. Any idea if the clusters/VM created as a part of integration test have an external IP?

I found this file where we test BPF NodePort but I don't see it using any external IP. Any idea if the clusters/VM created as a part of integration test have an external IP?

The tests should be run on GKE (https://jenkins.cilium.io/job/Ginkgo-CI-Tests-Pipeline-GKE/). I'd add a test case which checks whether NodePort service is accessible from inside a cluster by sending request to self InternalIP and ExternalIP + NodePort.

@brb It looks like the tests on GKE runs with kube-proxy only.  https://jenkins.cilium.io/job/Ginkgo-CI-Tests-Pipeline-GKE/2958/testReport/Suite-k8s-1/16/
Nodeport BPF test was skipped. Am I missing something?

I have created an initial commit that implements integration test. Let me know if this is a correct direction.

@AnishShah To me it looks like a correct direction. The following suite should be run on GKE:
SkipItIf(helpers.RunsWithoutKubeProxy, "Tests NodePort (kube-proxy)", func() {
			testNodePort(false, false, false, 0)
		})

I think on the GKE CI jobs, we have set kubeProxyReplacement=probe which should enable the host-reachable services (aka bpf_sock), so it should stress test your change.

Can we start GKE CI jobs?

test-gke

@AnishShah You can run it by yourself too: https://docs.cilium.io/en/v1.9/contributing/testing/e2e/#running-in-gke

All CI jobs including GitHub actions seem stuck. I'll try to close/reopen to trigger it. If that doesn't work, you might need to push again.

test-gke

test-gke

I was able to run the tests myself. When I ran the test against HEAD, it failed. When I ran the test with my fix, it passed.

The tests passed. Is this ready to be merged?

test-me-please

test-me-please

Can we re-run the failing test? I think it is a flake.

test-net-next

@joestringer Could you take another look?

test-me-please

can someone re-run test-net-next as it is flaky?

test-net-next

Proxy codeowners changes are negligible and I'm not worried about missing janitor codeowners given that we've had several reviews so we're bound to have looked at the code that's caught by janitor codeowner. CI all passing, good to merge.


<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED]

$ youtube-dl -v -i https://www.youtube.com/watch?v=d7n34bkF-Ew
[debug] System config: []
[debug] User config: ['--proxy', 'http://<IP_ADDRESS>:8087', '-o', '~/Downloads/%(title)s.%(ext)s', '-f', '22']
[debug] Command-line args: ['-v', '-i', 'https://www.youtube.com/watch?v=d7n34bkF-Ew']
[debug] Encodings: locale 'UTF-8', fs 'utf-8', out 'UTF-8', pref: 'UTF-8'
[debug] youtube-dl version 2014.03.12
[debug] Python version 3.3.5 - Linux-3.13.7-1-ARCH-x86_64-with-arch
[debug] Proxy map: {'http': 'http://<IP_ADDRESS>:8087', 'https': 'http://<IP_ADDRESS>:8087'}
[youtube] Setting language
WARNING: unable to set language: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:548)>
[youtube] d7n34bkF-Ew: Downloading webpage
ERROR: Unable to download webpage: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:548)>
  File "/usr/lib/python3.3/site-packages/youtube_dl/extractor/common.py", line 192, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/lib/python3.3/site-packages/youtube_dl/YoutubeDL.py", line 1173, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.3/urllib/request.py", line 469, in open
    response = self._open(req, data)
  File "/usr/lib/python3.3/urllib/request.py", line 487, in _open
    '_open', req)
  File "/usr/lib/python3.3/urllib/request.py", line 447, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.3/urllib/request.py", line 1289, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/lib/python3.3/urllib/request.py", line 1251, in do_open
    raise URLError(err)

I think it's related to python3, 'cause when I git cloned this repo and ran it with python2, everything just worked.

+1

For users who wish to opt out of certificate verification on a single connection, they can achieve this by providing the contextargument to urllib.urlopen :
import ssl

# This restores the same behavior as before.
context = ssl._create_unverified_context()
urllib.urlopen("https://no-valid-cert", context=context)

As a reference, you'll need --no-check-certificate anyway as GAE-based proxies, including GoAgent and XX-Net, regenerates all SSL certificates.


Bug: Query on date attribute doesn't work as expected

Given this page:
- First [date: 2024-07-08]
- Second [date: 2024-07-07]

Query:
item where date = '2024-07-08'
I'd expect that the query returns First. However, it does return no result at all. If I use the same query for selecting pages that contain a date as frontmatter attribute, it works as expected. Seems like selecting date attributes behaves differently than dates stored in other locations.

I just fixed this for frontmatter here: https://github.com/silverbulletmd/silverbullet/commit/2eb7e25854000090bdb3a8729aed3c3c1f06e381
Let me also apply that to inline attributes.

The issue here is that YAML parses strings of a date-like format as a JavaScript date, not as a string. A workaround is to put quotes around it: [date: "2025-08-01"]


Add a reloader controller to kubernetes deployment

We need to take advantage of initializing a rolling update if any gluu ConfigMap or secret is modified during a deployment.

@moabu did you mean re-deploying pods if any of config/secret updated?

yes I have tested a method and it is working . I hoping this will be in pygluu 1.04. This will be in the frontend @iromli.

In preparation for adding a reloader , configuration has been reduced to one file. We now have two ConfigMaps and one secret.


20151230-predict-income-from-census.md

why acs:

[x] it's well structured (cultivated) and has both regression and classification tasks
easily understood
[x] data set is not so large that requires extra infrastructure to handle
[x] motivated by https://www.kaggle.com/c/2013-american-community-survey
[x] footnote: race is a problematic term: https://en.wikipedia.org/wiki/Race_(human_categorization)
[x] use american community survey first since csv formatted:

[x] use 5 year for small populations https://www.census.gov/programs-surveys/acs/guidance/estimates.html
[x] acs vs census:

[x] https://www.census.gov/programs-surveys/acs/guidance.html
[x] http://www.prb.org/Publications/Articles/2009/differences.aspx

[x] why acs: https://www.census.gov/content/dam/Census/programs-surveys/acs/about/ACS_Information_Guide.pdf. About 1/38 households receives an invitation every year. These data are used to allocate federal funds.
[x] Under why census data?:
The census is a relatable dataset. Everyday people can understand census data because it's about them.
The census is an applicable dataset. The census has many use-cases from local and national government to market validation. (link to use-cases)
The census is a cultivated dataset. The census bureau goes to great lengths to ensure that the census is representative of the people it samples.
The census is a large dataset. There are many features in the dataset, which permits many questions that can be answered from it. (footnote: I avoid the term "big data" here because, for this blog post, the data is not "big data" since it and all operations on it fit within RAM.)
The census is a timeseries dataset. As a timeseries, the data can be used to predict current and future quantities.
With the above points in mind, I recognize that using census data can be problematic. Census data has been used to harm people (For examples of privacy concerns, see the ACLU's FAQ section about census data. https://www.aclu.org/frequently-asked-questions-national-census), and census data categorizes people by loaded terms, such as "race" (For reasons why "race" is a problematic term, see the Wikipedia article "Race (human categorization)". https://en.wikipedia.org/wiki/Race_(human_categorization).
For the sake of this exercise, I take the census data at face value. It's important to recognize that these aren't simply statistics; these are people.

About ACS:

[x] ACS docs:

[x] To access API: http://www.census.gov/data/developers/data-sets/acs-survey-5-year-data.html
[x] Use cases for ACS data: https://www.census.gov/programs-surveys/acs/about/survey-is-mandatory/acs-helps-communities.html
[x] Link to Kaggle scripts competition
[x] Instructions for ACS ftp: https://www.census.gov/programs-surveys/acs/data/data-via-ftp.html
[x] $3000, 8 weeks for custom table: https://www.census.gov/acs/www/data/custom-tables/request-a-custom-tab/
[x] ACS guidance: https://www.census.gov/programs-surveys/acs/guidance.html
[x] Use cases by data tool: https://www.census.gov/acs/www/guidance/which-data-tool/
[x] Tools chart: https://www.census.gov/programs-surveys/acs/guidance/which-data-tool/data-tools-chart.html
[x] When to use which year estimate: https://www.census.gov/programs-surveys/acs/guidance/estimates.html
[x] Handbooks: https://www.census.gov/programs-surveys/acs/guidance/handbooks.html
[x] From handbook App 1, use the dataset centered on each year with the largest window. This includes as many blocks as possible while keeping the data as recent as possible. Note that 1-year data is included in the 3-year estimate, which is included in the 5-year estimate. The data for each block also has its own confidence interval.
Give example following Handbook Appendix 1
[x] Useful handbooks:

[x] for general users with case studies: https://www.census.gov/content/dam/Census/library/publications/2008/acs/ACSGeneralHandbook.pdf
[x] for Public Use Micro Sample: https://www.census.gov/library/publications/2009/acs/pums.html
[x] for researchers (give link)

[x] Point-in-time estimates needs to have a shared baseline (e.g. 5-year)
[x] For hidden markov models for time series: (mcmc) http://nbviewer.ipython.org/github/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/blob/master/Prologue/Prologue.ipynb
[x] Include CPS data for income (from handbook)
[x] The purpose of this exercise is to provide actionable intelligence to enable a person to make a "data driven" decision, not to be an absolute authoritative measure of a quantity. The measure for good enough is making a testable prediction then testing it (i.e. hypothesis testing, "a/b testing").
[x] What is the American Community Survey? (link to main page, and kaggle competition)
from handbooks """
The American Community Survey (ACS) is the new
source for the information previously collected through
the decennial census long form. This information
includes topics such as income, employment status,
housing costs, and housing conditions. Unlike the
decennial census, ACS data are collected on a continuous
basis.
"""
About 250k addresses each year are requested to participate.
[x] Need to adjust dollar amounts for inflation.

[x] Related examples:

[x] https://mathematicaforprediction.wordpress.com/2014/03/30/classification-and-association-rules-for-census-income-data/
[x] http://www.knowbigdata.com/blog/predicting-income-level-analytics-casestudy-r
[x] https://rpubs.com/Jovin/census_data_income
[x] https://archive.ics.uci.edu/ml/datasets/Census-Income+(KDD) (See "Papers That Cite This Dataset")
[x] https://archive.ics.uci.edu/ml/datasets/US+Census+Data+(1990) (see "Papers That Cite This Dataset")

[x] From Public Use Micro Sample: https://www.census.gov/library/publications/2009/acs/pums.html

[x] Under "PUMS Geography":

[x] """
As noted earlier, to ensure the confi dentiality of ACS
respondents, PUMS fi les present data for a much more
limited set of geographic areas than the pretabulated
ACS products. PUMS fi les cannot be used to summarize
data for individual counties, cities, or other small
areas. It is possible to summarize data for the nation,
each of the states, the District of Columbia, Puerto
Rico, and areas known as Public Use Microdata Areas
(PUMAs).
As part of Census 2000, PUMAs were defined as areas
with 100,000 residents or more based on the populations
reported in Census 2000. The ACS uses these
same PUMAs.
"""
[x] "To show where a particular PUMA is located, the
Census Bureau has provided both maps and geographic
equivalency files."
[x] The Census 2000 PUMAs will be used until the
new PUMAs are delineated using 2010 Census counts.
[x] """ A separate geographic equivalency
fi le exists for each state. They are found in the
main PUMS link from the Census Bureau Web site:
http://www.census.gov/main/www/pums.html
For instance, using the New York equivalency fi le and
sorting the data by summary level code, one can see
which census tracts are grouped together in a PUMA or
which PUMAs compose New York County [Manhattan],
(03801-03810). Geographic Equivalency Files can be
accessed at the following FTP site: http://www2.census.gov/census_2000/datasets/PUMS/FivePercent/
The Missouri State Data Center created a tool that
allows PUMA users to enter the geography that they
are interested in to identify PUMA codes and
equivalent geographies. For more information, go to
http://mcdc2.missouri.edu/websas/geocorr2k.html
"""

[x] Under "Creating PUMS Tabulations"

[x] """For the 1-year PUMS the infl ation
adjustment variable (ADJUST) should be used to
produce income characteristics. The 3-year and 5-year
PUMS will carry two infl ation adjustment variables—
ADJINC for income applications and ADJHOUS for housing
cost applications. For additional information about
dollar-denominated data in the ACS, refer to Appendix 5."""
[x] """The advantage of going through
the ACS Web site is that the PUMS user verifi cation fi les
are listed. User verifi cation fi les provide estimates for
selected housing and population characteristics to help
data users determine that they are using the weights
correctly. """
[x] Download the data dictionary to understand the columns.
[x] The handbooks include statistical formulas for calculating standard error, margin of error and confidence intervals.
[x] Use replicate weights methods for errors since doesn't rely on distribution type.
[x] Use confidence intervals from numerical sampling instead of Z-score. Numerical sampling is more extensible.
[x] I choose not to weight by the coefficient of variation (Weight = 1/(coef_of_var)**2) but instead incorporate the uncertainty through numerical simulation. I prefer to sample the distributions to infer the statistical significance (reword, link to stats for hackers)
[x] Differentiate between Census Bureau data and census data
[x] Smallest area in PUMS is census block group
[x] Data from the Census Bureau balances representing areas and representing populations while also protecting respondents' privacy. Each record in the PUMS data is a group of census blocks that represents an area with about 100,000 people (and weighting factors?). Also, block groups with sparser populations are not included every year and are only represented in multi-year data sets.
[x] Much of the information in the handbooks is also replicated through the site.
[x] The handbook is from 2008, but only some of the links are out-of-date. The information is very thorough with many examples replete with use-cases.
[x] The margin of error calculations assume that the data is normally distributed, which often isn't the case for real-world data (an emperical, rank-based test for normally distributed data is z1,z2, astroML link)
[x] Consult "ACS Quality Measures" for nonsample error.
[x] For official poverty measures, see http://www.census.gov/hhes/

[x] Handbook for reasearchers https://www.census.gov/library/publications/2009/acs/researchers.html

[x] Complete "PUMS estimates for user verification"

[x] Design and methodology: https://www.census.gov/programs-surveys/acs/methodology/design-and-methodology.html
[x] Census Bureau's collection of reports and inforgraphics. : https://www.census.gov/programs-surveys/acs/library.All.html
[x] Why Census?

[x] I was inspired by this Kaggle competition: https://www.kaggle.com/c/2013-american-community-survey

[x] Predict all financial fields (q47 and q48)? Or just q48?
[x] Predict household income rather than personal: http://thenextweb.com/socialmedia/2012/01/24/the-top-30-stats-you-need-to-know-when-marketing-to-women/

About google cloud services:

[x] Storing data: https://cloud.google.com/docs/storing-your-data

[x] BigTable is for large applications
[x] Cloud Storage for immutable blobs
[x] BigQuery for online analytical processing.
[x] DataProc as a managed Hadoop and Spark service.
[x] Use DataLab when deploying analytics.

[x] Access the AppEngine application that runs the DataLab from datalab.cloud.google.com and Developers Console > Products & services > App Engine > Versions > click the version to access (e.g. "main (default)")

[x] Both BigQuery and Cloud Storage integrate with Google Prediction API.
[x] Compare to Google Prediction API.
[x] Use BigQuery when deploying to app. Use Postgres for individual development.

[x] Use Cloud Storage to save ACS data.

[x] Use gsutil to make a bucket in Cloud Storage. Requires to python 2.7. https://cloud.google.com/storage/docs/getting-started-gsutil
[x] Use stharrold.github.io as the bucket name: https://cloud.google.com/storage/docs/bucket-naming
[x] To verify site ownership: https://cloud.google.com/storage/docs/bucket-naming#domain-verification
[x] Use Durable Reduced Availability storage: https://cloud.google.com/storage/docs/storage-classes
[x] Cloud storage best practices: https://cloud.google.com/storage/docs/best-practices
[x] When scaling, use hadoop connector: https://cloud.google.com/hadoop/google-cloud-storage-connector
[x] As part of setup, ensure that the instance has "Read Only" access to "Storage": when creating a new instance, under "Project Access" > "Access and security"
[x] Rather than bucket, could just save file and make postgres db: https://cloud.google.com/solutions/setup-postgres-data-disk
[x] Use StringIO to read from Cloud Storage: http://pandas.pydata.org/pandas-docs/version/0.17.1/io.html#io-read-csv-table, https://cloud.google.com/storage/docs/gsutil/commands/cat, but gsutil requires python 2.7, so would have to download the file anyway. So need to make a spare disk anyhow.

ETL:

[x] Use own drive for data
[x] Read the data from ACS using urllib2.urllib(url).read()
[ ] out-of-core computations: http://nbviewer.ipython.org/github/jcrist/Dask_PyData_NYC/blob/master/Dask_DataFrame_Airline.ipynb
[ ] Get lat lon from zipcode: http://stackoverflow.com/questions/5585957/get-latlng-from-zip-code-google-maps-api
[ ] Get lat lon from group areas: http://www.census.gov/geo/maps-data/
[x] Download:

[x] http://www2.census.gov/programs-surveys/acs/data/pums/2013/5-Year/csv_hus.zip
[x] http://www2.census.gov/programs-surveys/acs/data/pums/2013/5-Year/csv_pus.zip
[x] http://www2.census.gov/programs-surveys/acs/tech_docs/pums/data_dict/PUMSDataDict13.txt
[x] Note: Data is mirrored to multiple locations within the ftp site. Files under http://www2.census.gov/programs-surveys/acs/  correspond to the sitemap of http://www.census.gov/programs-surveys/acs/

Useful sklearn sections:

[ ] sklearn sec 1: feat importance, partial dependence, prob of class belong
[ ] sklearn sec 2: manifold learning for vis; pca for vis and dim reduction
[ ] sklearn flowchart: http://scikit-learn.org/stable/tutorial/machine_learning_map/index.html; use as roadmap
[ ] sklearn sec 3:

[ ] k-fold validation for many k
[ ] link to "Note on shuffling"; use for cols and rows: http://scikit-learn.org/stable/modules/generated/sklearn.utils.shuffle.html
[ ] http://scikit-learn.org/stable/modules/grid_search.html use n_jobs=-1,  error_score=np.nan
[ ] http://scikit-learn.org/stable/modules/model_evaluation.html use dummy estimators for a baseline
[ ] use f1 score for classification, r2 for regression, compute probability of belonging to class with logistic regression and/or calibrating the prediction probability
[ ] save jobs with http://scikit-learn.org/stable/modules/model_persistence.html

[ ] Test distribution of test (evaluation) set and compare to multinomial distribution 9https://en.wikipedia.org/wiki/Multinomial_distribution): Link to Jake's stats for hackers in footnotes and "Helpful links"
[ ] sklearn docs sec 4:

[ ] Use Pipeline: http://scikit-learn.org/stable/modules/pipeline.html
[ ] Join linear and kernel pca using feature union
[ ] Examples of feature creation: http://scikit-learn.org/stable/modules/feature_extraction.html
[ ] Encoding categorical features: http://scikit-learn.org/stable/modules/preprocessing.html
[ ] Options for imputing missing values: http://scikit-learn.org/stable/modules/preprocessing.html
[ ] Custom transformers

[ ] sklearn docs section 6:

[ ] processing data at scale: http://scikit-learn.org/stable/modules/scaling_strategies.html#scaling-with-instances-using-out-of-core-learning
[ ] need to use incremental estimators

[ ] Also use combining estimators?
[ ] Kernel pca instead of pca?
[ ] Add sklearn docs to "Helpful links"
[ ] use warm_start for incremental random forest.
[ ] sklearn docs section 7:

[ ] give latency and throughput: http://scikit-learn.org/stable/modules/computational_performance.html#prediction-latency
[ ] give versions blas and lapack: http://scikit-learn.org/stable/modules/computational_performance.html#prediction-latency
[ ] Optimizing and profiling: http://scikit-learn.org/stable/developers/performance.html

[ ] create cross features since incremental pca is linear.
[ ] when to stop clustering: choose number of clusters with cross-validation. https://en.wikipedia.org/wiki/Determining_the_number_of_clusters_in_a_data_set
[ ] do feature selection based on explained variance
[ ] use classification report and confusion matrix for classification example: http://scikit-learn.org/stable/tutorial/statistical_inference/putting_together.html
[ ] example pipeline: http://scikit-learn.org/stable/tutorial/statistical_inference/putting_together.html
[ ] only use algorithms that currently implement a warm_start method for scalablility
[ ] Give confidence level for predictions
[ ] vectorize categories: http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.DictVectorizer.html#sklearn.feature_extraction.DictVectorizer
[ ] select features from model output: http://scikit-learn.org/stable/modules/feature_selection.html#feature-selection
[ ] Only choose methods that have feature importance criteria for actionable intelligence
[ ] Show which k-fold to use with a learning curve
[ ] AUC vs F1 for classification, doesn't matter but F1 is faster to compute: https://en.wikipedia.org/wiki/Receiver_operating_characteristic
[ ] Use R2 for regression: http://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html#sklearn.metrics.r2_score; diff metrics for diff purposes
[ ] To choose number of clusters:

[ ] Silhouette score: http://scikit-learn.org/stable/auto_examples/cluster/plot_kmeans_silhouette_analysis.html#example-cluster-plot-kmeans-silhouette-analysis-py
[ ] other ways: https://en.wikipedia.org/wiki/Determining_the_number_of_clusters_in_a_data_set

[ ] imputing with k-nearest neighbors:

[ ] https://github.com/scikit-learn/scikit-learn/issues/2989
[ ] show that imputing is better than other methods
[ ] R has many imputation methods

Helpful links:

[x] Helpful for a social scientists' perspective http://www-rohan.sdsu.edu/~gawron/python_for_ss/course_core/book_draft/data/PUMS_data.html
[x] For deciding r vs python http://programmers.stackexchange.com/questions/181342/r-vs-python-for-data-analysis

Other notes:

[x] use collapsable cells for code and images

Questions:

[x] What is the value-add?

[ ] relevant SO question: http://stackoverflow.com/questions/26564775/regex-to-parse-well-formated-multi-line-data-dictionary

moved many tasks to #36


[Language] Getting started with XML Parser in C# .

Topic Suggestion
Getting started with XML Parser in C#
Proposed article introduction
A software package or library that provides an interface or platform for a client application to work with XML documents is known as an XML parser. It guarantees that an XML document's layout is proper and approves such XML documents. In recent years, browsers have been built using XML parsers. To make the procedure easier, there are a few commercial tools on the market that may help with the separation or split of XML documents for a more productive output. The smaller parts of the XML text assist the user in comprehending each notion and efficiently applying it.
Key takeaways
1.How to create, open and read an XML file in C#
2. What is XML parsing in C#
3. How to do XML parsing in C# file
4. How to Read and Parse an XML File in C#
Article quality
This article will be a combination of theory and practical which will enable the reader to be able to work with XML documents  comfortably. The XML parser concept in C# will also be discussed in length.
References
N/A

There are several existing articles on this topic. How is yours unique?

@louisefindlay23
The existing articles lean a lot on the parsing concept. There are very few articles on reading and parsing of the files. Some of the articles in the sites are actually questions on how to go about the reading and parsing of the XML files. My article will be unique in that it will first cover the creating, opening and reading of XML files then cover the aspect of parsing. It will be covering almost everything in one article which I would say is an easier way of learning.

I'm going to close this topic due to inactivity.
It can be reopened.

Closed.

Just and FYI I will be closing this topic form just to help clear up the queue where possible - can be REOPENED at anytime when ready. 👍

Closing due to inactivity - can be reopened when article is ready to be linked.


v0.4 fails to compile

Trying to build the latest matterbridge master fails with this error:
$ go get github.com/42wim/matterbridge
# github.com/42wim/matterbridge/vendor/github.com/mattermost/platform/model
go-work/src/github.com/42wim/matterbridge/vendor/github.com/mattermost/platform/model/client.go:644: undefined: Compliance
go-work/src/github.com/42wim/matterbridge/vendor/github.com/mattermost/platform/model/session.go:30: undefined: TeamMember

It can be solved by copying mattermost/platform from matterbridge-plus.
go version 1.6.2

Whoops, had those locally, forgot to commit. Thanks!


[#471] Safely get line_number from PotentialSecret

This is a relatively naive approach to solving the issue in #471. I looked at the PotentialSecret class and it very clearly has a default value of 0 for line_number. But alas, intermittently it does not. So this should safely fall back to 0 if line_number isn't present.
Closes #471 (I think)

I'll review this one this weekend hopefully -- sorry for the delay.

Hey @js-truework, if you use the master branch, does it work?

Hey @js-truework, if you use the master branch, does it work?

I'll check it out. It'll be hard to say definitively until I've been able to run it a bunch of times, since it was an intermittent bug to begin with.
Thanks!


add arm support for install script

Currently the install script fallback to 386 unless the arch is x86_64.
As arm binaries are officially released, a simple case statement in the install script fix the issue.

Looks good to me @mirdhyn, thanks! :+1:


Compatibility with tokio_util::codec

I found this crate pretty helpful for running codecs when I would like to play without tokio. But currently there is missing some methods, which stops me to use it.
My pr #42 solves this issue.

Fixed with #42 thanks!


Add sorting algorithms in javascript

Issue Number
#1
Add insertion sort and bubble sort

Please read the CONTRIBUTING.md file and follow instructions there.
https://github.com/aniketsharma00411/algorithmsUse/blob/master/CONTRIBUTING.md


Add keycodes support

This is an initial attempt at adding keycodes support to DPF.
These allow to deal with keyboard keys in a layout-independent manner. It's particularly useful for piano-style widgets, to map keyboard keys to piano notes in the correct order.
This code is based on https://github.com/depp/keycode, especially the  <os>_tohid.c files and keycode.h in the src directory.
Tested with the piano widget on Linux, MacOS and Windows, with a couple of different layouts (Colemak, AZERTY, QWERTY).
The keycodes would be wrong when running the UI in Wine, but they would work fine on a real Windows 10 machine. I'm not sure if this is normal.
Perhaps it would be good to test this feature a little more before merging.

Looks good, but a few things to do:

rename COPYING into COPYING.keycode
make the keycode array static
make KEYCODE_EVDEV_OFFSET static too, and check if opcode < KEYCODE_EVDEV_OFFSET before doing the -= operation

there are a little oddities regarding spaces vs tabs, but I can fix them later myself
also, exposing the keycode API would be nice yes :)


Macro file output broken

Hi,
after running ./macphish.py -lh <IP_ADDRESS> -lp 443 -p osx/x64/meterpreter_reverse_https -a meterpreter -m
the .macro files in generated but the output seems broken. I can not even copy it so i had to make a screenshot since it only copies a small part of it.
I am running macphish on Kali Linux.

``

bump

Hey,
The tool only supports python payloads for now. Any of them would work.
Ps.Sorry for the late reply. I've been traveling abroad and my internet access has been limited.


For index

@charlesmike index pls

@charlesmike index pls

@amihaiemil thank you for the command, the indexing process started just now. I will let you know of the outcome shortly.

@charlesmike index

@charlesmike index

@amihaiemil thank you for the command, the indexing process started just now. I will let you know of the outcome shortly. In the meantime, you can check the logs here.

@charlesmike index

@amihaiemil Some steps failed when precessing your command. See logs for details.

@charlesmike index this page pls

@charlesmike index this page pls

@amihaiemil sorry I do not understand you. See this link for more details.

@charlesmike index this page

@charlesmike index this page

@amihaiemil the given page does not seem to belong to the website hosted in this repo!
Make sure that the link starts with http:// or https:// followed byyourusername.github.io/... Do not use the CNAME or any other link, even if it redirects toa page from this repo.
If this condition is met and it still doesn't work please open an issue here.

@charlesmike index this page

@charlesmike index this page

@amihaiemil thank you for the command, the indexing process started just now. I will let you know of the outcome shortly. In the meantime, you can check the logs here.

@charlesmike index this page

@amihaiemil Some steps failed when precessing your command. See logs for details.

@charlesmike index

@charlesmike index

@amihaiemil thank you for the command, the indexing process started just now. I will let you know of the outcome shortly. In the meantime, you can check the logs here.

@charlesmike index

@amihaiemil Some steps failed when precessing your command. See logs for details.

@charlesmike index

@charlesmike index

@amihaiemil thank you for the command, the indexing process started just now. I will let you know of the outcome shortly. In the meantime, you can check the logs here.

@charlesmike index

@amihaiemil Some steps failed when precessing your command. See logs for details.

@charlesmike index pls

@charlesmike index pls

@amihaiemil thank you for the command, the indexing process started just now. I will let you know of the outcome shortly. In the meantime, you can check the logs here.

@charlesmike index pls

@amihaiemil Some steps failed when precessing your command. See logs for details.

@charlesmike index

@charlesmike index

@amihaiemil thank you for the command, the indexing process started just now. I will let you know of the outcome shortly. In the meantime, you can check the logs here.

@charlesmike index

@amihaiemil Some steps failed when precessing your command. See logs for details.

@charlesmike index pls

@charlesmike index pls

@amihaiemil thank you for the command, the indexing process started just now. I will let you know of the outcome shortly. In the meantime, you can check the logs here.

@charlesmike index pls

@amihaiemil Some steps failed when precessing your command. See logs for details.

@charlesmike index

@charlesmike index

@amihaiemil thank you for the command, the indexing process started just now. I will let you know of the outcome shortly. In the meantime, you can check the logs here.

@charlesmike index

@amihaiemil Some steps failed when precessing your command. See logs for details.

@charlesmike index

@charlesmike index

@amihaiemil sorry I do not understand you. See this link for more details.

@charlesmike index

@charlesmike index

@amihaiemil thank you for the command, the indexing process started just now. I will let you know of the outcome shortly. In the meantime, you can check the logs here.

@charlesmike index

@amihaiemil Some steps failed when precessing your command. See logs for details.

@charlesmike index

@charlesmike index

@amihaiemil thank you for the command, the indexing process started just now. I will let you know of the outcome shortly. In the meantime, you can check the logs here.

@charlesmike index

@amihaiemil Some steps failed when precessing your command. See logs for details.

@charlesmike index

@charlesmike index

@amihaiemil thank you for the command, the indexing process started just now. I will let you know of the outcome shortly. In the meantime, you can check the logs here.

@charlesmike index

@amihaiemil Some steps failed when precessing your command. See logs for details.

@charlesmike index pls

@charlesmike index pls

@amihaiemil thank you for the command, the indexing process started just now. I will let you know of the outcome shortly. In the meantime, you can check the logs here.

@charlesmike index pls

@amihaiemil Some steps failed when precessing your command. See logs for details.

@charlesmike index

@charlesmike index

@amihaiemil thank you for the command, the indexing process started just now. I will let you know of the outcome shortly. In the meantime, you can check the logs here.

@charlesmike index

@amihaiemil Some steps failed when precessing your command. See logs for details.

@charlesmike index

@charlesmike index

@amihaiemil thank you for the command, the indexing process started just now. I will let you know of the outcome shortly. In the meantime, you can check the logs here.

@charlesmike index

@amihaiemil Some steps failed when precessing your command. See logs for details.

@charlesmike index

@charlesmike index

@amihaiemil thank you for the command, the indexing process started just now. I will let you know of the outcome shortly. In the meantime, you can check the logs here.

@charlesmike index

@amihaiemil Some steps failed when precessing your command. See logs for details.

@charlesmike index

@charlesmike index

@amihaiemil thank you for the command, the indexing process started just now. I will let you know of the outcome shortly. In the meantime, you can check the logs here.

@charlesmike index

@amihaiemil Some steps failed when precessing your command. See logs for details.

@charlesmike index

@charlesmike index

@amihaiemil thank you for the command, the indexing process started just now. I will let you know of the outcome shortly. In the meantime, you can check the logs here.

@charlesmike index

@amihaiemil Some steps failed when precessing your command. See logs for details.

@charlesmike hello there

@charlesmike hello there

Hi, @amihaiemil! I can help you index your Github-hosted website, provided that the repository name respects the format owner.github.io (or has a gh-pages branch) and the command is given by the repo's owner.

@charlesmike test

@charlesmike test

@amihaiemil sorry I do not understand you. See this link for more details.

@charlesmike hi


onboard: Fix gobject-introspection

Motivation for this change
https://github.com/NixOS/nixpkgs/issues/56943
Onboard was not working under xmonad (AFAIK it works fine under gnome).
Fixed as per ticket #56943 by adding gobject-introspection to buildInputs (it's already in nativeBuildInputs.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

Notify maintainers
cc @johnramsden

I've fixed this in #67915 along with some other issues with onboard.


Fix HTML tag auto-completion in code blocks.

Found that the Razor compiler has some really interesting quirks when it comes to finding the "owner" of an HTML element when within code blocks. Added some "ensurement" methods to workaround these quirks until the compiler can fix them.
Added tests.

Before

After

Fixes dotnet/aspnetcore#33901

Merging in to for first 17.0-P2 update. Feel free to still review and i'll update after-the-fact


Tokenizer issue on Google Collab

Python code:
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("./T5_Paraphrase_Paws", use_fast=False)
model = AutoModelForSeq2SeqLM.from_pretrained("./T5_Paraphrase_Paws")

sentence = "This is something which i cannot understand at all"
text =  "paraphrase: " + sentence + " </s>"
encoding = tokenizer.encode_plus(text,pad_to_max_length=True, return_tensors="pt")
input_ids, attention_masks = encoding["input_ids"].to("cuda"), encoding["attention_mask"].to("cuda")

outputs = model.generate(
    input_ids=input_ids, attention_mask=attention_masks,
    max_length=256,
    do_sample=True,
    top_k=200,
    top_p=0.95,
    early_stopping=True,
    num_return_sequences=5
)

for output in outputs:
    line = tokenizer.decode(output, skip_special_tokens=True,clean_up_tokenization_spaces=True)
    print(line)

Error on Collab:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-21-c0b69f8e2f6a> in <module>()
      1 from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
      2
----> 3 tokenizer = AutoTokenizer.from_pretrained("./T5_Paraphrase_Paws", use_fast=False)
      4 model = AutoModelForSeq2SeqLM.from_pretrained("./T5_Paraphrase_Paws")
      5

4 frames
/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_fast.py in __init__(self, *args, **kwargs)
     94         else:
     95             raise ValueError(
---> 96                 "Couldn't instantiate the backend tokenizer from one of: "
     97                 "(1) a `tokenizers` library serialization file, "
     98                 "(2) a slow tokenizer instance to convert or "

ValueError: Couldn't instantiate the backend tokenizer from one of: (1) a `tokenizers` library serialization file, (2) a slow tokenizer instance to convert or (3) an equivalent slow tokenizer class to instantiate and convert. You need to have sentencepiece installed to convert a slow tokenizer to a fast one.

Solution tried but unsuccessful: set use_fast to False
tokenizer = AutoTokenizer.from_pretrained("./T5_Paraphrase_Paws", **use_fast=False**)

Alright,
pip install sentencepiece

And restarting runtime seems to do the job.
Cheers,

Alright,
pip install sentencepiece

And restarting runtime seems to do the job.
Cheers,


Create Album Modal

As a user, I want to be able to click on a track and have a modal come up with all tracks from the album so i can add more to the queue.

Global modal --> function that dispatchs with props
e.g.
openAlbumModal = (albumId) => EventHorizon.dispatch(‘OPEN_ALUBM_MODAL’, {
   albumId
})

in the action, set open=true
subscribe contained in the modal code with composeWithTracker


example readme updates and gitignore sync

…and anything else you can think of before first publish!

Circle tests failing seems to have to do with npm vs yarn but will have to investigate further, for now we need these updates!

Config was missing, sorted now #13


Reduce MIN_SIZE for BFV moduli

The current setting of 30 bits does not support the ring dimension of 1024. It should be reduce to a value in the range of 15-20.

Set MIN_SIZE to 14 as 12289 is the first prime that is congruent to 1 mod 2*512 (512 is the smallest secure ring dimension)


[bug] CORS regression when tauri.conf.json > build > devPath points to a directory.

Describe the bug
I updated to v1.2.0 and now I'm getting this CORS error (js console).
[Error] Origin http://<IP_ADDRESS>:1430 is not allowed by Access-Control-Allow-Origin. Status code: 403
[Error] Fetch API cannot load http://<IP_ADDRESS>:5001/api/v0/id due to access control checks.
[Error] Failed to load resource: Origin http://<IP_ADDRESS>:1430 is not allowed by Access-Control-Allow-Origin. Status code: 403 (id, line 0)
[Error] Unhandled Promise Rejection: TypeError: Load failed
	(anonymous function) (index.d004a4cf.js:39453)
	(anonymous function)

The app is connecting to a local go-ipfs daemon on port 5001 using ipfs-http-client and it worked fine in previous versions.
I've had devPath pointed to a directory for a long time, since I was never able to get past CORS serving on localhost.
I tried various things in tauri.conf.json > tauri.allowlist.http and security.csp but nothing helped. Maybe I just didn't find the magic combo? If there's come combination of things I could do to get it working via dev server, that would be cool too.
Reproduction
git clone https://github.com/iohzrd/identia.git
npm run update
npm start

Expected behavior
no error
Platform and versions
> tauri info

Environment
  › OS: Manjaro 22.0.0 X64
  › Node.js: 16.18.0
  › npm: 8.19.2
  › pnpm: Not installed!
  › yarn: Not installed!
  › rustup: 1.25.1
  › rustc: 1.65.0
  › cargo: 1.65.0
  › Rust toolchain: stable-x86_64-unknown-linux-gnu

Packages
  › @tauri-apps/cli [NPM]: 1.2.0
  › @tauri-apps/api [NPM]: 1.2.0
  › tauri [RUST]: 1.2.0,
  › tauri-build [RUST]: 1.2.0,
  › tao [RUST]: 0.15.2,
  › wry [RUST]: 0.22.0,

App
  › build-type: bundle
  › CSP: unset
  › distDir: ../dist
  › devPath: ../dist
  › framework: Svelte

App directory structure
  ├─ src
  ├─ node_modules
  ├─ dist
  ├─ src-tauri
  ├─ scripts
  ├─ public
  ├─ .vscode
  └─ .git

Stack trace
No response
Additional context
I also tried tweaking the IPFS CORS config, but I don't think that's the problem, since it worked before.
My app automatically configures the IPFS daemons "Access-Control-Allow-Origin" and "Access-Control-Allow-Methods" , so that shouldn't be the issue. See resulting config below.
  "API": {
    "HTTPHeaders": {
      "Access-Control-Allow-Methods": [
        "GET",
        "POST",
        "PUT"
      ],
      "Access-Control-Allow-Origin": [
        "http://<IP_ADDRESS>:5001",
        "tauri://localhost",
        "https://tauri.localhost"
      ]
    }
  },
...

You'll need to add http://<IP_ADDRESS>:1430 to the Access-Control-Allow-Origin array since that's the URL we use now when the devPath is pointed to a local directory. We added this to add HMR support in this case.

That was one of the first things I tried, but it didn't work, so I tried a bunch of things and was pulling my hair out. Turns out there was an orphan/rogue IPFS process that was getting in the way. Sorry for the erronous report.


Support XMB

https://github.com/bnu-wangxun/Deep_Metric

I would like to contribute to this issue.
One way to implement the Cross-batch-memory (XBM) module is to extend MetricLoss with
a memory data structure and necessary enqueue and dequeue functionality. It is also
important to enable a warmup period where XBM is not active.
In the following implementation example, the memory data structure consists of a tf.Variable
with dynamic size in the 0th dimension, while enqueue and dequeue are implemented with
tf.concat and slicing respectively.
from typing import Callable, Optional, Union

import tensorflow as tf

from tensorflow_similarity.distances import Distance, distance_canonicalizer
from tensorflow_similarity.losses import MetricLoss
from tensorflow_similarity.losses.multisim_loss import multisimilarity_loss
from tensorflow_similarity.types import FloatTensor

def _add_memory_variable(tensor):
    """Creates an empty variable with same shape and dtype as `tensor`."""
    shape_no_batch = tensor.shape[1:]
    dtype = tensor.dtype
    init_value = tf.constant([], shape=[0, *shape_no_batch], dtype=dtype)
    var = tf.Variable(
        initial_value=init_value,
        shape=[None, *shape_no_batch],
        dtype=dtype,
        trainable=False,
    )
    return var

class XBM(MetricLoss):
    def __init__(
        self,
        fn: Callable,
        memory_size: int,
        warmup_steps: int = 0,
        reduction: Callable = tf.keras.losses.Reduction.AUTO,
        name: Optional[str] = None,
        **kwargs
    ):
        super().__init__(fn, reduction, name, **kwargs)

        if isinstance(fn, MetricLoss):
            self.distance = fn.distance

        self.memory_size = memory_size
        self.warmup_steps = warmup_steps
        self._y_true_memory = None
        self._y_pred_memory = None
        self._total_steps = tf.Variable(0, dtype=tf.int64, trainable=False)

    def call(self, y_true: FloatTensor, y_pred: FloatTensor) -> float:
        if self._y_true_memory is None:
            self._y_true_memory = _add_memory_variable(y_true)
        if self._y_pred_memory is None:
            self._y_pred_memory = _add_memory_variable(y_pred)

        # enqueue (concat batch to beginning of memory)
        y_true_mem = tf.concat([y_true, self._y_true_memory], axis=0)
        y_pred_mem = tf.concat([y_pred, self._y_pred_memory], axis=0)

        # dequeue (truncate to memory size limit)
        y_true_mem = y_true_mem[: self.memory_size]
        y_pred_mem = y_pred_mem[: self.memory_size]

        def _xbm_step():
            # update memory
            self._y_true_memory.assign(y_true_mem)
            self._y_pred_memory.assign(y_pred_mem)
            return y_true_mem, y_pred_mem

        def _warmup_step():
            return y_true_mem, y_pred_mem

        y_true_mem, y_pred_mem = tf.cond(
            self._total_steps > self.warmup_steps,
            true_fn=_xbm_step,
            false_fn=_warmup_step,
        )
        self._total_steps.assign_add(1)

        return super().call(y_true_mem, y_pred_mem)

This implementation can be used either as a wrapper around existing MetricLoss loss instances:
loss = MultiSimilarityLoss(distance="cosine")
loss = XBM(loss, memory_size=100, warmup_steps=1000)
model.compile("adam", loss=loss)

or as a parent class to inherit XBM functionality:
class XBMMultiSimilarityLoss(XBM):
    def __init__(
        self,
        memory_size: int,
        distance: Union[Distance, str] = "cosine",
        alpha: float = 1.0,
        beta: float = 20,
        epsilon: float = 0.2,
        lmda: float = 0.5,
        warmup_steps: int = 0,
        name: str = "XBMMultiSimilarityLoss",
    ):
        distance = distance_canonicalizer(distance)
        self.distance = distance

        super().__init__(
            multisimilarity_loss,
            name=name,
            reduction=tf.keras.losses.Reduction.NONE,
            distance=distance,
            alpha=alpha,
            beta=beta,
            epsilon=epsilon,
            lmda=lmda,
            memory_size=memory_size,
            warmup_steps=warmup_steps,
        )

loss = XBMMultiSimilarityLoss(distance="cosine", memory_size=100, warmup_steps=1000)
model.compile("adam", loss=loss)

This trains just fine
Epoch 1/5
1000/1000 [==============================] - 18s 16ms/step - loss: 0.8051  # warmup steps
Epoch 2/5
1000/1000 [==============================] - 16s 16ms/step - loss: 1.8837
Epoch 3/5
1000/1000 [==============================] - 16s 16ms/step - loss: 1.7174
Epoch 4/5
1000/1000 [==============================] - 16s 16ms/step - loss: 1.5286
Epoch 5/5
1000/1000 [==============================] - 16s 16ms/step - loss: 1.3939

However, there is one problem with the current Distance class in order to correctly support XBM.
Right now the call method of every Distance has the following signature:
def call(self, embeddings: FloatTensor) -> FloatTensor:

This only allows computing the distances between the embeddings themselves like d(embeddings, embeddings),
where d is some mathematical distance function. Therefore the current XBM implementation can only compute
the distances between the memory embeddings themselves like so d(memory_embeddings, memory_embeddings).
The XBM paper you referenced requires the distances to be computed between batch embeddings and memory embeddings like so
d(batch_embeddings, memory_embeddings). So a refactoring of the Distance class might be necessary in order to
properly support XBM.
We have to consider if the call method of Distance classes should support computing distances between embeddings
and some optional target_embeddings like so:
def call(self, embeddings: FloatTensor, target_embeddings: Optional[FloatTensor] = None) -> FloatTensor:
    if target_embeddings is not None:
        return d(embeddings, target_embeddings)
    else:
        return d(embeddings, embeddings)

or perhaps like this:
def call(self, embeddings: Union[FloatTensor, Tuple[FloatTensor, FloatTensor]]) -> FloatTensor:
    if isinstance(embeddings, tuple):
        embeddings, target_embeddings = embeddings
        return d(embeddings, target_embeddings)
    else:
        return d(embeddings, embeddings)

Please let me know what you think!

Hi @chjort, thanks for working on this! I like the implementation, especially how this supports using the existing MetricLoss classes.
Regarding the Distance signature, I think I prefer adding a second parameter as an Optional[FlaotTensor] over the more complex Union of types. Alternatively, we could explicitly require both embedding inputs. This would be more verbose, but would avoid the conditional and make it clearer which two embeddings we are comparing. One downside is this would be a breaking change.
def call(self, source_embeddings: FloatTensor, target_embeddings: FloatTensor) -> FloatTensor:
     ...

@ebursztein wdyt?

I think it's a reasonable change if we document it carefully. I would be more comfortable if we also think through how this will work with prototype network e.g SWAV to make sure we only change the signature once.
Make sense?

Comparison between two different sets of embeddings is central in XBM, but also in a multitude of self-supervised contrastive approaches including:

SvAV
Barlow Twins
SimCLR
BYOL
SimSiam

Many of these self-supervised approaches have specialized loss functions which do not expect any lables, but instead computes a loss from the embeddings only (often by comparing embeddings from different views (augmentations) of a batch).
Changing Distance to require two embeddings will allow these self-supervised losses to also utilize the Distance class, and not having to re-implement the distance function within the self-supervised loss. Essentially expanding the set of use-cases Distance can support.
I will start working on a Pull Request where I refactor the Distance class to have the following call signature:
def call(self, query_embeddings: FloatTensor, key_embeddings: FloatTensor) -> FloatTensor:

and make the necessary changes up stream to support all current functionality of the master branch (mostly the losses module is affected here). If changes to the Distance class gets merged, it will make sense to continue working on XBM.

The refactoring of distances and losses is now proposed in PR #222, making it easier to implement XBM in a new future pull request if #222 gets merged.


PoC: .write(Buffer) support, related to #173

mashed up quick support for .write(chunk: Buffer), works rather well as long as input is a Buffer, in case it is a utf8 string performance degradation is quite bad. See #173 for why it might be needed
if raw buffer is something that is worth exploring further then I'd be happy to put more effort into having fast paths for both types of encoding, probably by preselecting mode of operation at the time of SonicBoom instantiation
benchSonic*1000: 2.189s
benchSonicSync*1000: 8.190s
benchSonic4k*1000: 2.147s
benchSonicSync4k*1000: 1.827s
benchCore*1000: 2.561s
benchConsole*1000: 4.835s
benchSonicBuf*1000: 840.388ms
benchSonicSyncBuf*1000: 6.882s
benchSonic4kBuf*1000: 824.605ms
benchSonicSync4kBuf*1000: 481.976ms
benchCoreBuf*1000: 824.466ms
benchSonic*1000: 2.105s
benchSonicSync*1000: 8.275s
benchSonic4k*1000: 2.179s
benchSonicSync4k*1000: 1.811s
benchCore*1000: 2.220s
benchConsole*1000: 5.287s
benchSonicBuf*1000: 856.635ms
benchSonicSyncBuf*1000: 6.852s
benchSonic4kBuf*1000: 849.188ms
benchSonicSync4kBuf*1000: 480.347ms
benchCoreBuf*1000: 802.708ms

Added contentMode setting. Not sure if the approach I've selected is the best from maintenance standpoint, but benchmarks are ok
benchSonic*1000: 428.597ms
benchSonicSync*1000: 6.054s
benchSonic4k*1000: 408.553ms
benchSonicSync4k*1000: 240.762ms

benchCore*1000: 2.785s
benchConsole*1000: 4.752s

benchSonicBuf*1000: 468.216ms
benchSonicSyncBuf*1000: 6.271s
benchSonic4kBuf*1000: 458.785ms
benchSonicSync4kBuf*1000: 343.907ms
benchCoreBuf*1000: 1.081s

benchSonic*1000: 407.736ms
benchSonicSync*1000: 6.123s
benchSonic4k*1000: 403.239ms
benchSonicSync4k*1000: 236.555ms

benchCore*1000: 2.768s
benchConsole*1000: 5.307s

benchSonicBuf*1000: 455.687ms
benchSonicSyncBuf*1000: 6.225s
benchSonic4kBuf*1000: 454.69ms
benchSonicSync4kBuf*1000: 334.973ms
benchCoreBuf*1000: 1.079s

Master branch perf for comparison, so utf8 mode (default one) seem to be unaffected by changes
benchSonic*1000: 452.573ms
benchSonicSync*1000: 6.064s
benchSonic4k*1000: 429.246ms
benchSonicSync4k*1000: 258.853ms
benchCore*1000: 2.722s
benchConsole*1000: 4.725s

benchSonic*1000: 422.106ms
benchSonicSync*1000: 6.054s
benchSonic4k*1000: 426.696ms
benchSonicSync4k*1000: 253.348ms
benchCore*1000: 2.724s
benchConsole*1000: 5.305s

Only downside I see is that I have copied non-buffer related functions and changed logic there, so if any logic changes in the future then there are multiple places where this needs to be addressed

@mmarchini could you take a look as well?

Sorry, was drowned in GitHub Notifications for months. I'll take a look at it this week.

overall looks fine, I didn't review the tests yet. Main concern as you mentioned is code duplication, so it'll be up to the project to decide if taking the burden of having some logic implemented in two places is worth it for this feature.
As for the performance aspects of this, I can see concat being faster than multiple writes, although that'll be highly dependent on system specs. Can you share specs for the machine where you ran your benchmarks?

Chip Apple M1 Max
Memory 64 GB

Updates incoming, in terms of bench
Node 20 performs ~25% better, Node 18 ~ 10%
node 20.3.1
benchSonic*1000: 411.93ms
benchSonicSync*1000: 6.231s
benchSonic4k*1000: 403.094ms
benchSonicSync4k*1000: 231.349ms
benchCore*1000: 1.900s
benchConsole*1000: 6.654s
benchSonicBuf*1000: 479.036ms
benchSonicSyncBuf*1000: 6.643s
benchSonic4kBuf*1000: 479.021ms
benchSonicSync4kBuf*1000: 338.352ms
benchCoreBuf*1000: 647.737ms
benchSonic*1000: 402.104ms
benchSonicSync*1000: 6.381s
benchSonic4k*1000: 411.276ms
benchSonicSync4k*1000: 230.133ms
benchCore*1000: 1.879s
benchConsole*1000: 7.073s
benchSonicBuf*1000: 488.614ms
benchSonicSyncBuf*1000: 6.739s
benchSonic4kBuf*1000: 468.49ms
benchSonicSync4kBuf*1000: 346.683ms
benchCoreBuf*1000: 657.002ms

node 18.16.0
benchSonic*1000: 435.906ms
benchSonicSync*1000: 6.272s
benchSonic4k*1000: 437.36ms
benchSonicSync4k*1000: 252.272ms
benchCore*1000: 2.252s
benchConsole*1000: 4.260s
benchSonicBuf*1000: 799.471ms
benchSonicSyncBuf*1000: 6.771s
benchSonic4kBuf*1000: 796.44ms
benchSonicSync4kBuf*1000: 645.533ms
benchCoreBuf*1000: 893.629ms
benchSonic*1000: 418.545ms
benchSonicSync*1000: 6.380s
benchSonic4k*1000: 417.882ms
benchSonicSync4k*1000: 245.55ms
benchCore*1000: 2.214s
benchConsole*1000: 5.374s
benchSonicBuf*1000: 787.246ms
benchSonicSyncBuf*1000: 6.849s
benchSonic4kBuf*1000: 822ms
benchSonicSync4kBuf*1000: 638.927ms
benchCoreBuf*1000: 888.464ms

@mmarchini please take a look if the changes are sufficient based on your comments

wow, quite a speedup also on writing things compared to core.

Would be interesting to bench this with io_uring support as fs operations should be much faster on linux
Some weirdness is going on with win/node-14 combo during dep install

Some weirdness is going on with win/node-14 combo during dep install

GHA is broken on that platform, just exclude it in the workflow file.

I'll try to take a look over the weekend :)

Would be interesting to bench this with io_uring support as fs operations should be much faster on linux

I have a Linux machine I can try it on. Will report back with the results

anything I should do with this to move it forward? :)