readMedia is dropping bytes when reading a binary file (e.g. jpeg)

I successfully uploaded a jpeg file as an attachment but when trying to retrieve it via the readMedia call it was coming back smaller than it went in.
I was able to successfully download the attachment media via Postman so I knew it was correctly uploaded.
In digging through the code I think I found the issue in the createRequestObject (line 49 in request.js).  When handling the response data event it appears that bytes are being dropped when appending the chunk to the data string.  My guess is that there are some bytes that make a string unhappy and either the string is being truncated or the characters are being dropped.
To test the theory I modified the code response.on('data' to read as follows:
    var data = "";
    var length = 0;
    var lastChunk = null;
    response.on("data", function(chunk) {
        data += chunk;
        length += chunk.length;
        lastChunk = chunk;
    });

data.length - length was always growing throughout the processing of the response.

I changed the MediaReadMode in the connection policy to Streamed and handled the response directly.  This resolved my issue and may be the best way to handle binary files, however this should be noted in the documentation.

@limeybastow thank you for reporting this. I think this should work in any MediaReadMode meaning that's a bug.
I suspect the root cause is what you guessed and there are some bytes that make a string unhappy.
Are you interested in provide a fix patch for this? I will merge your patch as your contribution if you do so.

I've been pondering how it might be fixed without making a breaking change for anyone who's currently successfully using it to return text documents.  One thought I had was to look at the content type and if it was not some sort of text file (text, html, xml etc.) to return a node Buffer object instead of a string.

@ato9000 @rnagpal any suggestion on this?

I was looking at the .NET version of our SDK and it seems the Media(that represents the attachment content) is of Stream type, so looks like we always return a stream which can be converted based on the content-type. So I'm fine returning a node Buffer object(if that's the equivalent of Stream in Node) for all cases and let the user handle it.

@rnagpal - A node Buffer isn't a stream.  It's just a byte array.  In the node version of the SDK you can set the MediaReadMode to either Streamed or Buffered when you connect.  If you set it to Streamed it returns a stream that works fine with binary files.  If you set it to Buffered it's currently returning a string.  I ended up just change the mode to Streamed and that worked fine for me so I wouldn't have a problem with having the Buffered dropped completely and thus obsolete the MediaReadMode option.
If we wanted to keep the Buffered mode it should probably be converted to a node Buffer instead of a string or have an option to choose one or the other but in either case it would create a breaking change.

We have recently announced deprecation version 1.x of the Azure Cosmos JavaScript SDK. We will end support for the documentdb package and this repo on August 30, 2020. Please update to our new package @azure/cosmos as soon as possible. If you encounter any issues, you can raise them in the Azure central JS SDK repo. If something is preventing you from upgrading to the latest version of the SDK, you can always email me directly<EMAIL_ADDRESS>


Forward dqlite logging

This branch configures dqlite to use lxd's logger instead of its default one (from the stdlib)

jenkins: test this please


Bumping to Rugged v0.23.0b1

:pray:

Thanks @arfon! :turtle: :palm_tree: :nail_care:

Thank you sir!


Handing Centos setup in bootstrap.yml

Resolves: https://github.com/Islandora-CLAW/CLAW/issues/749
Depends on https://github.com/Islandora-CLAW/CLAW/issues/746
This PR sets up bootstrapping for Centos systems.  It skips installing python and aptitude, and uses a different list of packages.  I'm replacing build-essential with equivalent packages available to yum, and am removing some that don't exist without adding extra repos.
You can test it with
ISLANDORA_VAGRANT_DISTRO="centos/7" vagrant up --no-provision
ansible-playbook -i inventory/vagrant -t bootstrap playbook.yml

It works, but I'm sure it can be done better and am open to suggestions.

WORK!!!

I don't have a free Redhat/Centos box sitting around. But this looks fine and if @DigitLib says its good then I'm good. 👍


feat(js-client)!: Segregation of responsibility between js-client packages [fixes DXJ-525]

What have been done

Js-client doesn't care about aqua types anymore, interface simpler
added validation to builtins and public API
various bug fixes and optimizations

CLI PR with passing tests
https://github.com/fluencelabs/cli/pull/630


Feature: Simulation Presets

The simulation would be more interactive if it allowed the user to create their own simulation presets.

Simulation Presets

name - gui text input
saving - json file, by pressing a gui button
loading - from a json file (by name)

Entity Creation

radius - gui slider
position - selected by clicking on the screen
speed - by dragging while left mouse button down

Implemented in https://github.com/scorp18pl/Barnes_Hut_Algorithm/commit/c911460717145b77c8614e58caa683ccfa6ed82a.


APT whitelist request for clang-5.0 in Precise

Supercedes https://github.com/travis-ci/apt-package-whitelist/issues/3296 and others.

Thanks @BanzaiMan


Scale link shortening

Currently using Netlify's _redirects to store links that are shortened. This makes it to where people need a github account to add links, which is not ideal.
Some proposed solution:

Collect links to be shortened via a form and use Netlify's serverless functions to re-direct at the CDN edge until build is completed. Some open questions:

How can we make sure the service isn't being used for non-blm related info without the PR review process in place? Auditing the links could be an answer.
How do we make sure a re-direct isn't clobbering a markdown page being used as an info directory?

Host a form that interacts with an SCM bot that opens a PR for reviewers to confirm a re-direct add. Can use webhooks for notifications.

Will probably require a good bit of help reviewing but necessary in either scenario.

I might hold off on some of the bail resources specifically because another open source repo is handling that (redirected their currently)

It’s deployed via Netlify rn because it’s pretty great for automatic builds and DNS related stuff

Possibly, i’m open to suggestions here!

I agree with @illegalprime on the form+pr bot. Can be thrown together quick with bootstrap and express, would maybe need some error checking for dupes and invalid chars.
I've never worked with Netlify but looking into it.

@wipeyadocsoff this was implemented by #11 , we should close this.


Limit total number of entities to return

Calculate the number of possible entities returned from a query (similar to GitHub's GraphQL explorer) and fail if the number exceeds a limit defined by an environment variable.

This check should be done in the graphql crate. The best place to do this is probably in fn execute_root_selection_set, in graphql/src/execution/execution.rs. That function receives a SelectionSet. Since we don't currently support fragments, that simplifies the logic, we can recursively dive into the selection set fields (which may also contain selection sets) and calculate the number of entities the query might return. The calculation should be done as described in the github API docs.

Assigning it back to @leodasvacas since I got stuck with other stuff.. Sorry Leo. I'll be studying Rust on the side for sure, and will be able to help with future (smaller) tasks

Requirements

Each query should have a maximum complexity, calculated based on the amount of entities returned, as described in the github API docs.
If a query exceeds the complexity, it should not be executed and instead return an error.

Proposed solution

Check the complexity in execute_root_selection_set, before executing the query. Walk the AST selection set tree to calculate the complexity, using the first argument.
Set the complexity limit as an env var, defaulting to no limit.

Tasks

[ ] Implement the solution in graph node.

@leodasvacas A couple comments:

Doing it in execute_root_selection_set feels ok for queries but for subscriptions, it would be better if we caught this at the time the subscription is set up and then reject the subscription from the start. So I'd prefer for this to be a standalone function that we run either in GraphQlRunner or in execute_query / execute_subscription before executing a query and/or subscription.

About the first requirement: amount of entities returned is almost accurate. It's more potential amount of entities returned, which is roughly estimated using the first arguments + one-to-one relationships.


How can I show pointer dynamically on specific point by writing code?

As per requirement, I want to show the pointer at a specific position without tapping the graph on the load of the screen. How can I achieve this?

Hi @kr396 👋🏼
You can use dataPointLabelComponent and related props.
See documentation
See props table

Hi @kr396 👋🏼 You can use dataPointLabelComponent and related props.
See documentation
See props table

Hi, @ak-spotter ,
Thanks for the quick reply.  But, I need a pointer always on. Currently, it shows once the user taps on the graph and vanishes based on pointerVanishDelay. Also, need to show that pointer on specific point on the graph load. Like, show a pointer on the current month once the graph loads. After that user can move that to view different months' data.

@kr396 The example above shows the pointer always on. The code is also in the documentation link shared above.
Add the properties dataPointLabelComponent  and showStrip to the data item you want the pointer to be always on.

@Abhinandan-Kushwaha @ak-spotter
How can I set point visibility to always? It shows once the user clicks or scrolls on the graph. I'm not able to find any property in pointer config

@kr396 Don't use pointerConfig, use dataPointLabelComponent instead.
Add the properties dataPointLabelComponent  and showStrip to the data items you want the pointer to be always on.

@kr396 Don't use pointerConfig, use dataPointLabelComponent instead.
Add the properties dataPointLabelComponent and showStrip to the data items you want the pointer to be always on.

Thanks for the quick response. Can you please help me with this - My requirement is, that I have to show the current month's pointer on the graph load. After that user can see data of any month by scrolling. That's why I'm using a pointer.
Thanks in advance.

@kr396 Please see the example code on this link
Open the link and scroll down to this-

Just copy the code and run it. It solves your use case.

@kr396
I know you want to show pointer but still no need to use pointerConfig
Adding dataPointLabelComponent and showStrip inside the objects of data array will add a pointer to the specific data points. And the pointer will not disappear automatically.
Comment here if it doesn't help.

@kr396 I know you want to show pointer but still no need to use pointerConfig Adding dataPointLabelComponent and showStrip inside the objects of data array will add a pointer to the specific data points. And the pointer will not disappear automatically. Comment here if it doesn't help.

@Abhinandan-Kushwaha
Using this I can show pointer to specific point. But it is not scrollable. I cannot move/scroll strip/pointer.

Hey @kr396 👋
Sorry for my late response and the inconvenience due to a misunderstanding on my side.
If I understood correctly, your exact requirement is to show the pointer at a specific position, and then the user should be able to reposition the pointer by touching or scrolling the chart.
Thanks for requesting this feature. I have now added this feature in version 1.3.17
You just need to add the property initialPointerIndex inside the pointerConfig object.
Version 1.3.17 onwards, the following properties are also configurable inside the pointerConfig object-

initialPointerAppearDelay (number)
persistPointer (boolean)

Hey @kr396 👋 Sorry for my late response and the inconvenience due to a misunderstanding on my side.
If I understood correctly, your exact requirement is to show the pointer at a specific position, and then the user should be able to reposition the pointer by touching or scrolling the chart.
Thanks for requesting this feature. I have now added this feature in version 1.3.17 You just need to add the property initialPointerIndex inside the pointerConfig object.
Version 1.3.17 onwards, the following properties are also configurable inside the pointerConfig object-

initialPointerAppearDelay (number)
persistPointer (boolean)

Thanks, This is exactly what I wanted. Thank You @Abhinandan-Kushwaha


[DRAFT] Add Datastore layer

Signed-off-by: Max Lambrecht<EMAIL_ADDRESS>Pull request check list

[X] Proper tests/regressions included?
[X] Documentation updated?

Affected functionality
Added a new Datastore implementation that supports Postgres
Description of change

This PR adds migration SQLs to create the schema based on the new data model, and also adds the SQL Queries.
The interfaces, model structs, and lower level DB operations were auto generated using sqlc which uses the SQL files from migrations and queries as input.
A new Datastore struct was added, that defines CRUD methods for all the entities of Galadriel.
When the Datastore is created and it connects to the configured DB, the schema is validated and migrations applied if needed, using golang-migrate.
Tests for the Datastore were implemented using dockertest, which starts a container from a Postgres image, runs the migrations, and then run the test.

Which issue this pull requests fixes

Great stuff! Did a first quick pass-through and got stuck thinking into two different things:
On a more high-level discussion, do you still see we having one interface to abstract the datastore layer, in which each of the different implementations (postgres, mysql, etc) will have to satisfy? I couldn't find that interface anymore, but the Datastore object looks like it would implement this interface.
and the 2nd topic is related to using a connection pool, can be discussed in the comment below.

About the abstract datastore layer, I think that is this one: querier.go

Great stuff! Did a first quick pass-through and got stuck thinking into two different things:
On a more high-level discussion, do you still see we having one interface to abstract the datastore layer, in which each of the different implementations (postgres, mysql, etc) will have to satisfy? I couldn't find that interface anymore, but the Datastore object looks like it would implement this interface.
and the 2nd topic is related to using a connection pool, can be discussed in the comment below.

About the abstract datastore layer, I think that is this one: querier.go

We could still abstract the Datastore layer in an interface that uses entity objects from the API. I didn't think about that because this PR introduces a Datastore that just supports one dialect, and I still don't know if we'll need that abstraction anyway, even in the case of supporting multiple dialects.
The querier interface is auto-generated by the sqlc and it's a lower level interface that uses DB models.

We could still abstract the Datastore layer in an interface that uses entity objects from the API. I didn't think about that because this PR introduces a Datastore that just supports one dialect, and I still don't know if we'll need that abstraction anyway, even in the case of supporting multiple dialects.
The querier interface is auto-generated by the sqlc and it's a lower level interface that uses DB models.

Right, it makes sense. Yes, the querier interface looks good enough, at least for now. Although this higher level abstraction will still be implemented in some form, (at the handlers, most likely?) by having an interface we're better segregating this logic for our handlers. Anyways, not something for right now

LGTM!! Btw, you're not planning on incorporating these changes into galadriel on this PR, right? If not, I'm ready to approve.

No need to approve just yet, I'm expecting more feedback from the rest of the team.

Btw, you're not planning on incorporating these changes into galadriel on this PR, right?

Just realized my last question was a bit ambiguous, I wanted to ask about replacing the current In memory datastore to use this implementation. This should happen in a separate PR right?

Btw, you're not planning on incorporating these changes into galadriel on this PR, right?

Just realized my last question was a bit ambiguous, I wanted to ask about replacing the current In memory datastore to use this implementation. This should happen in a separate PR right?

IMO migrating Galadriel to the new Datastore is not worth doing right now, since the whole implementation of Galadriel will be mostly re-visited while moving from the POC to the MVP. Each new (re)implemented functionality in Galadriel Server will need to use the new Datastore.

Amazing work @maxlambrecht ❤
Did a code pass-through and it looks impeccable, I'm just not approving it now cause I didn't have time to test yet.
One topic that came to my mind though, is that I noticed we're not supporting TLS for the postgres connection. IDK if this is something needed for now, I don't think we've discussed this yet, but I'm pretty sure we'll have to think about this at some point. WDYT?

Thanks Juliano! ❤
The Postgres connection string allows TLS, something like:
"sslmode=require sslrootcert=server-ca.pem sslcert=client-cert.pem sslkey=client-key.pem hostaddr=your_host port=5432 user=your_user dbname=your_db"


Meter filter view

Fixes # (issue)
Changes
Please provide a brief description of the changes here.
For significant contributions please make sure you have completed the following items:

[ ] CHANGELOG.md updated for non-trivial changes
[ ] Unit tests have been added
[ ] Changes in public API reviewed

Codecov Report

Merging #83 (b68b221) into main (3a9d087) will decrease coverage by 0.63%.
The diff coverage is 61.95%.

@@            Coverage Diff             @@
##             main      #83      +/-   ##
==========================================
- Coverage   92.99%   92.36%   -0.62%
==========================================
  Files         196      198       +2
  Lines        7038     7114      +76
==========================================
+ Hits         6544     6570      +26
- Misses        494      544      +50

Impacted Files
Coverage Δ

...ntelemetry/sdk/metrics/view/attributes_processor.h
100.00% <ø> (ø)

sdk/include/opentelemetry/sdk/metrics/view/view.h
66.67% <0.00%> (+3.04%)
:arrow_up:

sdk/src/metrics/meter_context.cc
36.37% <0.00%> (-1.73%)
:arrow_down:

sdk/test/metrics/sync_metric_storage_test.cc
100.00% <ø> (ø)

sdk/src/metrics/meter.cc
9.10% <4.77%> (-9.09%)
:arrow_down:

sdk/test/metrics/multi_metric_storage_test.cc
68.43% <68.43%> (ø)

...telemetry/sdk/metrics/state/multi_metric_storage.h
84.62% <84.62%> (ø)

...ntelemetry/sdk/metrics/state/sync_metric_storage.h
52.95% <100.00%> (ø)

...clude/opentelemetry/sdk/metrics/sync_instruments.h
100.00% <100.00%> (ø)

sdk/src/metrics/sync_instruments.cc
100.00% <100.00%> (ø)

... and 2 more


Failed to install minishift

On my Mac I'm trying to start gofabric with minishift. But it fails to install minishift. Maybe because  minishift has changed it naming scheme with v1.0.0-beta.1 ?
https://github.com/minishift/minishift/releases/download/v0.9.0/minishift-darwin-amd64 => https://github.com/minishift/minishift/releases/download/v1.0.0-beta.1/minishift-1.0.0-beta.1-darwin-amd64.tgz
Error message:
$ gofabric8 start --minishift --vm-driver=virtualbox
fabric8 recommends OSX users use the xhyve driver
xhyve driver already installed
Downloading https://github.com/jimmidyson/minishift/releases/download/v1.0.0-beta.1/minishift-darwin-amd64...
Downloaded /Users/mdl/.fabric8/bin/minishift
Downloading https://storage.googleapis.com/kubernetes-release/release/v1.5.1/bin/darwin/amd64/kubectl...
Downloaded /Users/mdl/.fabric8/bin/kubectl
oc is already available on your PATH
using the executable /Users/mdl/.fabric8/bin/minishift
Unable to get status fork/exec /Users/mdl/.fabric8/bin/minishift: exec format error

$ head /Users/mdl/.fabric8/bin/minishift
<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="Content-type" content="text/html; charset=utf-8">
    <meta http-equiv="Content-Security-Policy" content="default-src 'none'; base-uri 'self'; connect-src 'self'; form-action 'self'; img-src data:; script-src 'self'; style-src 'unsafe-inline'">
    <meta content="origin" name="referrer">
    <title>Page not found · GitHub</title>
    <style type="text/css" media="screen">
      body {
        background-color: #f1f1f1;

$ gofabric8 version
gofabric8, version 0.4.112 (branch: 'unknown', revision: 'homebrew')
  build date:       '20161225-08:57:37'
  go version:       '1.7.4'

$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.11.3
BuildVersion:	15D21

same issue when attempting to use beta. Move off beta and all seems well.

Updated gofrabic8 from v0.4.112 to v0.4.113. Works now.

same issue. version update did not help on mac
[INFO] gofabric8> Unable to get status fork/exec /Users/bahadyr/.fabric8/bin/minishift: exec format error

I've not tried fabric8 on minishift lately as there was an issue with CORS https://github.com/minishift/minishift/issues/257  I think that until that error is resolved we need to stick to minishift version: 0.9.0 or use minikube in the meantime.


BGR for the win

aha, for the PNG fix:
./lvgl/src/lv_misc/lv_color.h
typedef union
{
struct
{
uint8_t blue;
uint8_t green;
uint8_t red;
uint8_t alpha;
} ch;
uint32_t full;
} lv_color32_t;
and a bunch of other structs are BGR.
to fix the hue shift just change the order
typedef union
{
struct
{
uint8_t red;
uint8_t green;
uint8_t blue;
uint8_t alpha;
} ch;
uint32_t full;
} lv_color32_t;
refactor should be done here because of the BGR default assumption, this means that this will mess all the renders and a new LV_COLOR_BGR should be added to lv_conf to treat the displays that really are BGR.

I think you are expected to handle this within your display driver.

not really, all colors should be RGB default then you should handle the BGR in your display or wherever you want.
Assuming all displays are BGR by default is illogic and the official blog post proves it.

Let's ask @kisvegabor how he thinks this should be handled.

look, this is not my library and don't care that much, I found my fix, keep it the way it is, but it seems really interesting since Gabor did the png load blog post and did not notice the BGR/RGB issue on the pc sim. almost all image formats are RGB and you expect that if you configured properly your display that you should get the proper image displayed, and you shouldn't have to dig around the code to find the issue, at least fix the fb and monitor to make the BGR to RGB conversion.
I get that this is MCU oriented hence the BGR but at least treat the rest properly.
@embeddedt - just my 2 cents, you are really hostile in your approach, polish those soft skills, same aggression as the using 2 framebuffers issue I posted earlier that you closed.

@embeddedt
"By default the LittlevGL assumes RGB order, however it can ba swapped by setting LV_SUBPX_BGR  1 in lv_conf.h." - HAHAHAHAHAH
@embeddedt - slow clap, really nice my friend. See I knew there was something I read, the fonts are treated as normal RGB with a BRG fix as the display should.
My advice to you, stop moderating and start reading the documentation.

@RRPRIME I'm sorry; my goal was not to come across as being hostile or demeaning. Sorry if that's the impression you got. I do try to keep up-to-date with everything but sometimes I forget about a feature/implementation detail.
By the way, the only reason your other issue got closed is because it was a question, which (per project policy) should be directed to the forum and not to our issue tracker.

@RRPRIME
@embeddedt is clearly the greatest contributor of LittlevGL. He helps a lot of people here and in the forum who are very grateful for him.  Fortunately, our user base is growing and therefore the number of questions is also growing. We do the support and development in our free time and not always have time to give long, super kind answers.
Besides, you can't expect anybody (including me) to know all the features or immediately see the connection between a question and a seemingly unrelated feature. (It's really unrelated, I'll explain below)
Finally, it seems you are the one who should improve those soft skills. Telling the most active contributor what he should do ("stop moderating and start reading the documentation") - after open 2 issues - is really rude and we don't welcome this style here.

Let's talk about the original question hopefully in a more pleasant style.

may the people struggle with the BGR !!!

I don't understand this. Are the other colors correct in your display or is it any issue only for PNG images?

"By default the LittlevGL assumes RGB order, however it can ba swapped by setting LV_SUBPX_BGR 1 in lv_conf.h."

It's related only to fonts. Subpixel rendered fonts are 3 times wider and the color of every color channel (R,G,B) is described. LV_SUBPX_BGR can be used if the order of the color channels physically inside a pixel is in BGR order. It might be the same or different from in which order the display needs the color information. (E.g. after rotating the by 180° the physical order of color channels is swapped but the display still needs the pixels in the original format.).

Hi,
I'm using lvgl 6.1 and freeRTOS on a LPC1788 (olimex MOD-LCD4.3" board). I use the 2 full size buffer method for the display driver and a touche pad.
Everything is looking really nice and working greate. But... my LCD (or LCD peripheral ?) is BGR and not RGB. I searched how to change that in the lcd peripheral but didn't found anything. Olimex just deal with it by swaping red and blue during pixel writing. But I cannot use the write_pixel callback with the 2 full size buffer.
I have tried to add LV_SUBPX_BGR 1 in the lv_conf.h file, but it do not the trick. Maybe it is only available on version 7.x of lvgl ?
Thank's for the work anyway 😃

Is it working if you swap red and green in lv_colot_t and the relates LV_COLOR_SET/GET macros?
What is your color depth? I can help to make it work and even add this as a config option in  lv_conf.h``

Thank you for the quick reply. My color depth is 32 bit.
I've tried to swap red and blue in lv_colot_t but it does not the trick (certainly because I use 32 bit depth). The macro LV_COLOR_SET/GET only get the channel .red and .blue so I've not tried to swap them.
I've joined my lv_conf.h and my porting in case you need more information.
lvgl_conf_and_port_olimex_mod_lcd.zip
At the moment, to handle the color swap I'm using the 1st buffer method. I declare 3 screen size buffers. The first is used by lvgl to render the graphics. The 2 others are used by the LCD. When flush_cb is called by lvgl I "copy and swap red and blue" of the area from the lvgl render buffer (1st) to the hidden lcd buffer. When the copy is finished, I simply swap the lcd pointer (like the 3rd buffer method).
Because of the buffer area copy, it's not as efficient as possible. Because lvgl doesn't always render all the screen when it calls flush_cb, I guess I cannot use DMA.
But the performance when I use the first buffer method seems more than enough. Even when I tell the task that call lv_task_handler to sleep periodically for 10ms, it shows everything smoothly and render the animations beautifully. That means people here have done a really good job 👍
When using freeRTOS, I use 2 task like that:
/**
 * @brief update lvgl tick
 */
void update_lvgl_tick()
{
    while(1)
    {
        lv_tick_inc(1);
        vTaskDelay( 1 / portTICK_PERIOD_MS );
    }
}

/*
* @brief littlevGL update screen task
*/
void littlvGL_screen_update()
{
    while(1)
    {
        lv_task_handler();
        // vTaskDelay( 10 / portTICK_PERIOD_MS ); // just to test lvgl speed
        // no vTaskDelay is called because we use the lowest prio
        // and systick is at 1ms
    }
}

That means the fewer other tasks are working, the better the render will be.

I've also played with it and concluded the same: the easiest thing is to modify LV_COLOR_MAKE.


Update 2-lambda-function.md

Changed link to securing Lambda function to actual page

Thanks for submitting this PR.  The link has been fixed as part of the tutorial update.


Error on sqlserver.js

Hi Peter,
I guess, there is a parameter mismatch on the SqlBuilder function of sqlserver.js.
The variable referenced in the code as "agent" but defined as "aget" in the parameter list.
function SqlBuilder(skip, take, aget) {
this.agent = agent;
this.builder = [];
this._order = null;
this._skip = skip >= 0 ? skip : 0;
this._take = take >= 0 ? take : 0;
this._set = null;
this._define;
this._fn;
this._join;
this._fields;
this._schema;
this._primary;
this._group;
this._having;
this._is = false;
this.hasOperator = false;
}
Regards,
Ozgen

Hi,
many thanks! This is my big mistake. I don't use MS SQL Server :-( so therefore there was the bug. I have fixed it.
Thank you!


First biweekly blog post of the CC Catalog Visualization project - GSoC 2019

This PR corresponds to the first biweekly blog post of the CC Catalog Visualization project - GSoC 2019
Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

Looks good, @soccerdroid!


What's meaning of bi-directional model ?

In https://github.com/Wang-Tianwei/Decoupled-attention-network/issues/1#issuecomment-569848838 issue,
you mentioned that the performance of paper could be get from bi-directional model.
I have checked the accuracy of various test dataset and accuracies were lower than paper.
So I want to try the best condition of models.
Is "bi-directional model" related with DTD module?
But "pre_lstm" in DTD is already using "bidirectional=True" flag.
I'm confusing if we can apply "bidirectional=True" with nn.GRU instead of nn.GRUCell(your code).
It looks nn.GRUCell is proper method because DTD of DAN have to check character of each step.

"Bi-directional model" means double CAM-DTD modules, one for forward prediction (reading order) and the other for reverse prediction. And the final result is determined by which direction yields higher probability.

okay, it make sense.
In paper, you mentioned that "We use the bi-directional decoder proposed in (Shi et al. 2018) for final prediction"
Thank you :)


EnableInvariantCultureFallback ignored?

Hi Valdis,
I was wondering if you could provide some insight into what I'm trying to achieve with the localization provider.
I have a requirement where I need to query the provider for localized labels in the fallback languages configured for the current page in EPiServer.
So for instance, if the fallbacks are defined as en-US => en-GB => en-CA
I want to first see if there is a label for en-US and if not test if one exists for en-GB.
However when I call GetString(expression, en-US) I always get a label back even though there isn't one configured - it looks like it is falling back to the invariant culture of EN, in which there is a label.
I have tried setting EnableInvariantCultureFallback to false on initialize but it still appears to fallback.
ConfigurationContext.Setup(ctx => {                 ctx.EnableInvariantCultureFallback = false; });
Do you have any ideas? Looking at the decompiled provider code it looks like it should return null but its not.
Thanks again
Paul

hi,
resource translations follow not only their on configuration (invariant fallback) but also is respecting episerver settings. can you check your episerver localization in web.config file. I'd that one is configured to fallback - it will fallback when you ask for translation.
So essentially - DbLocalizationProvider is just one of the providers in episerver's list. Episerver is asking all providers one by one which has the translation for given resource for given language. If DbLocalizationProvider returns null (for specific language) Episerver will "catch" it and following localization settings will "try" to find translation in next round bit this time using next fallback language.
Overall it's quite complex if you go deeper there.
I'm not sure whether there is option to "check" if translation exists for given language using provider API for Episerver. One trick would be to not use LocalizationService (Episerver's one) but LocalizationProvider (directly from DbLocalizationProvider namespace)..
Please follow up on this. If we are missing this API - think adding would be quite simple.

Hi Valdis,
So I took a further look through your provider code, but unfortunately still can't find a way around this.
I'll break down the important calls that are made from your provider, through to the localization service itself.
So an example I am working with currently is a label being used on the cart page:
string message = new GetTranslation.Query("Cart.Apply", language, false).Execute();
The language provided to this query is en-US, and I'm specifying no fallbacks (I've also removed fallbacks from my config aswell).
Your execute method looks fine - it's exactly how I want it to behave - try get the label using the provided culture, and if one isn't found and we enable fallbacks, use the invariant culture:
public string Execute(GetTranslation.Query query)
      {
        LocalizationService instance = ServiceLocator.Current.GetInstance<LocalizationService>();
        string stringByCulture = instance.GetStringByCulture(query.Key, query.Language);
        if (string.IsNullOrEmpty(stringByCulture) && instance.FallbackBehavior.HasFlag((Enum) FallbackBehaviors.FallbackCulture) && query.UseFallback && (object.Equals((object) query.Language, (object) instance.FallbackCulture) || object.Equals((object) query.Language.Parent, (object) CultureInfo.InvariantCulture)))
          return this._originalHandler.Execute(new GetTranslation.Query(query.Key, CultureInfo.InvariantCulture, false));
        return stringByCulture;
      }

It calls GetStringByCulture in the LocalizationService. GetStringByCulture if I follow it then does the following:
CultureInfo actualFallbackCulture = this.GetActualFallbackCulture(fallbackBehavior, culture);
      string localizedString;
      if (this.TryGetStringByCulture(resourceKey, normalizedKey, culture, actualFallbackCulture, out localizedString))
        return localizedString;
      return this.GetMissingFallbackResourceValue(fallbackBehavior, fallback, resourceKey, culture);

GetActualFallbackCulture will give us the invariant culture for my culture I've provided (en) and use this in the TrygetStringByCulture call just below it.
So it seems like I will always get a value back from the localization service regardless of if I specify no fallbacks in the provider.
It seems like an impossible situation using the scenario above - do you know of any other way I can maybe bypass the localization service altogether? It looks like, for EN cultures anyway, I would always get a value back, and have no way to know if the fallback language I've provided has an actual localized label, or if its the result of the invariant culture being used.
Thanks again,
Paul

hi,
sorry for delay here. yes, seems like there is a missing piece to get precise results back when querying for no-existing language translation and explicitly specifying no fallback to be used.
I'll see what we can do here..

hi,
couple follow-ups:

is there a reason why you are using GetTranslation.Query and not LocalizationService?
can you paste your <localization> element from <episerver.framework>?

did I get it correctly that in your case you have to follow fallback languages configured for specific page and not for the framework (via <>localization> element in web.config)?

Hi,
I tried both the GetTranslation.Query and LocalizationService aswell to be honest, out of curiosity. But I found both will call the same underlying GetStringByCulture method in the LocalizationService, which causes my issue.
Here is my localization configuration:
<localization fallbackBehavior="Echo, MissingMessage">             <providers>                 <add name="db" type="DbLocalizationProvider.EPiServer.DatabaseLocalizationProvider, DbLocalizationProvider.EPiServer" />                 <add virtualPath="~/Resources/LanguageFiles" name="languageFiles" type="EPiServer.Framework.Localization.XmlResources.FileXmlLocalizationProvider, EPiServer.Framework.AspNet" />             </providers>         </localization>
Yeah, exactly - per page I need to follow fall backs configured if any, and test to see if there is a localized label in that language.
Thanks,
Paul

ok, and do you need Echo & MissingMessage fallbacks for the framework?

no, its not crucial for us to have them.

hi,
sorry for delay here. so basically this functionality is somehow limited due to fact how Episerver is behaving. In order for library to be "good citizen" in the platform, it has to respect Episerver's rules. Which means that library respects what settings Episerver has configured.
So by having
<episerver.framework>
    <localization fallbackBehavior="Null">
      <providers>
          ....
      </providers>
    </localization>
</episerver.framework>

With fallback set to null you should be able to receive back null when requesting resource translation for missing language:
// _provider is injected LocalizationProvider
var r = _provider.GetStringByCulture(() => Class1.JustSimpleResource, new CultureInfo("fr"));

Hi,
Sorry for the delay in responding.
I did try out your suggestion but unfortunately it didn't return null for me when providing an en based culture - so for instance en-US, if I don't have a label configured for en-US it just returns me EN back, so I can never fallback to other EN based languages like en-GB from en-US.
I see your point though - its down to the way in which the localisation service falls back to the invariant culture of the culture you provide to it, internally.
Time to think of some alternatives! Thanks for your help.

That's very strage. I got this behaviour in raw AlloyTech site. Can share sample code if that helps..

I guess your example is slightly different to mine in that you requested for FR culture, which is an invariant culture.
If you test with, for example en-US, having no label configured for en-US, you should see that it will default to EN and pass you back a value for that label instead, because of this line in the localization service:
CultureInfo actualFallbackCulture = this.GetActualFallbackCulture(fallbackBehavior, culture);
You may have already tried this though :)

it shouldn't matter either you are asking in specific culture (with region) or general. if you do have Null set as behavior for the localization service in web.config -> you should get back null. Just tested once again..

Unfortunately I don't get the same result.
And you have a value for the label in the invariant culture? So EN in this case.
The only reason I ask is that from what I can tell it boils down to this method:
public virtual string GetStringByCulture(string resourceKey, FallbackBehaviors fallbackBehavior, string fallback, CultureInfo culture)
    {
      if (string.IsNullOrEmpty(resourceKey))
      {
        if (fallback != null)
          return fallback;
        if (fallbackBehavior.HasFlag((Enum) FallbackBehaviors.Echo))
          return resourceKey;
        if (fallbackBehavior.HasFlag((Enum) FallbackBehaviors.Null))
          return (string) null;
        Validator.ThrowIfNullOrEmpty(nameof (resourceKey), resourceKey);
      }
      Validator.ThrowIfNull(nameof (culture), (object) culture);
      string[] normalizedKey = this.NormalizeKey(resourceKey);
      CultureInfo actualFallbackCulture = this.GetActualFallbackCulture(fallbackBehavior, culture);
      string localizedString;
      if (this.TryGetStringByCulture(resourceKey, normalizedKey, culture, actualFallbackCulture, out localizedString))
        return localizedString;
      return this.GetMissingFallbackResourceValue(fallbackBehavior, fallback, resourceKey, culture);
    }

Which will always look to get the ActualFallbackCulture - which results in EN for a query of en-US for me.
If you have no value for EN either then it will return null.
I'll test this on Alloys also to make sure I haven't missed anything.
Thanks again!

yes, you are correct. I misunderstood setup. Yes, if you will have no regional culture translation, but there will be one for "parent" culture, you will get that one. actual method "guilty" for this is protected virtual bool TryGetStringByCulture(...)

however, IMHO trying pretty hard it would be possible to mimic this behavior and replace localization service completely.

Yeah, it's proving kind of impossible to do what I need to do.
It seems a little confusing however that within the localization service it would fallback like this, and kind of ignore our fallback settings we have provided via config.
Anyway, thank you for your time on this! It seems there is no way around the localization service logic.


Rabbitmq scaler: Allow providing separate paremeters for user and password

Proposal
I would like to be able to provide username/password as separate parameters for the RabbitMQ scaler, in order to be able to get host from Environment variable and username/password from separate environment variables/secrets
Use-Case
We are using single RabbitMQ deployment for multiple applications which have separate secrets containing their credentials. Since credentials are generated by helm templates we have no way to know what they will be in advance. Adding to this we are using podpreset to inject the environment variables containing the username and password, which means we cannot combine everything in 1 variable using the pod manifest, because the username/password variables are injected at the end of the variables array. Having the option to reference separate secret/environment variables from which username and password to be taken is crucial for such deployments.
Anything else?
No response

I strongly second this. Another obvious use case is using the Rabbit MQ Message Topology Operator to generate users. This operator produces secrets with separate username, password data. Additionally, you need to fetch the host, amqpPort, vhost, etc from another secret produced by the Rabbit MQ Cluster Operator.
Providing each of these as separate parameters to the TriggerAuthentication would allow us to simply point to the existing secrets, instead of having to devise a crazy way to combine them all into a URL on yet another secret.

I think this makes sense, WDYT @JorTurFer ?

yes, IMO it totally makes sense.
Are you willing to contribute with this @wasabii @iordaniordanov ?

I would like to tackle this issue

it's yours

What about this feature? Any ETA? We have deployment to multiple namespaces using kustomize and without ability of providing each of these as separate parameters to the TriggerAuthentication we can't simply point to the existing secrets generated by RabbitMQ Cluster Operator and/or RabbitMQ Messaging Topology Operator. Using hostFromEnv with something like this inside example-deployment definition:
envFrom:
  - prefix: RABBITMQ_
    secretRef:
      name: example-deployment-default-user
env:
  - name: RABBITMQ_ConnectionString
    value: amqp://$(RABBITMQ_username):$(RABBITMQ_password)@$(RABBITMQ_host):$(RABBITMQ_port)

also does not work because keda sees these variables before expansion, although it works inside example-deployment's container just fine.

@xoanmm

@b1czu I'm working in the PR associated, I think that I'll be able to finish in the following weeks, I'm having a hard time since I haven't programmed anything in go for a few weeks

Is there an approx timeline on this feature? This is one that's important to us too

AFAIK, the PR was abandoned, so you can open another PR adding the support if you are willing to contribute @zargarf

This feature would be particularly useful when using the secrets generated from the RabbitMQ Operator

@xoanmm , are you still interested on contributing with this? If you cannot any other folk can tackle it

Any update regarding this feature. We're facing a similar situation, and it would be great to have the username and password as separate fields.

There isn't any news AFAIK. Are you willing to continue with the feature?


Added: Persian Locale

This PR will add Persian language locale (fa/fa_IR) for Ghost

[x] The build will pass (run yarn test:all and yarn lint)

@royalfig hey there, sorry for the mention, can you accept the workflow?


Does not work on Mac

There's a problem running the create_module.sh script on Mac.
This seem to be due to version missmatch of SED. When running the same command in an Ubuntu Docker container it works fine.
$ bash -c "$(curl -sL https://raw.githubusercontent.com/roramirez/MagicMirror-Module-Template/master/create_module.sh)"
MagicMirror installation not found.
Please input its path now (or restart script in its directory): /path
Insert your module name: MyModule
Do you want create in /path/module/MyModule (y/N) y
Pickup a license
  1. MIT (Default)
  2. ISC
Choice: 1
Type a short description of what your module does (leave empty to insert a Todo)
: Todo
Cloning into '/var/folders/s9/tkvlqcy57w37x46r69d2x9b40000gn/T/tmp.jUQrYWPk'...
remote: Counting objects: 160, done.
remote: Total 160 (delta 0), reused 0 (delta 0), pack-reused 160
Receiving objects: 100% (160/160), 22.42 KiB | 0 bytes/s, done.
Resolving deltas: 100% (82/82), done.

I'm afraid that answer is a bit complex,
There's an issue that you cannot really update sed, instead you can install gsed (gnu sed) and alias it to sed. That's about as close as it gets, but that's not always working either, especially in shell scripts etc.
http://stackoverflow.com/questions/37639496/how-can-i-check-the-version-of-sed-in-os-x
usage: sed script [-Ealn] [-i extension] [file ...]
       sed [-Ealn] [-i extension] [-e script] ... [-f script_file] ... [file ...]

Tried a few alterations.
This seem to solve the problem for Mac users:
http://stackoverflow.com/questions/7573368/in-place-edits-with-sed-on-os-x
sed -i '' s/\{\{MODULE_NAME\}\}/$MODULE_NAME/g

Please send a Pull Request and I can review it.

Hi @MarcusSjolin
Did you have any update about this?

Fixed  by #17


CLOUDSTACK-8601. VMFS storage added as local storage can be re-added …

…as shared storage.
Fail addition of a VMFS shared storage pool in case it has already been added as local storage in CS.

Aside from a couple minor comments, LGTM. :)

Who wants to step in and finish this work? It seems the original author is not able to finish it. If no one steps in, we'll have to close the PR without merging it so please help :-).

The code LGTM. Are we under the assumption that it was properly tested? If so, I can just merge it at this point.

@mike-tutkowski Thanks! It seems the testing / verification is still to do and that is the work. Merging itself can be done with a one-liner. Are you able to verify the fix in your lab?

Hey...I just noticed something.
I have a datastore that I manually added to my ESXi hosts (shared storage) that is based on an iSCSI target.
When I restart the CS MS, this datastore (which I already added to CloudStack as shared primary storage) gets re-added as local storage (so it's listed as both shared and local primary storages).
I think this is perhaps what lead to this pull-request in the first place, but the problem still exists.
What happens now that this pull-request code is in is that the datastore conflict is detected and an error message is printed, but I still see my shared primary storage has been added as well as local storage.
I'm looking into this more, but wanted to point out this observation.
I had forgotten about this issue (I think it's pretty old). At the time, we put in a workaround where datastores that started with "-" or "" were assumed to be shared and not re-added as local. It just happens that my manually added, shared datastore does not start with "-" or "", so I see the problem.

OK, I mainly remembered the issue correctly, but not entirely. The way we avoid this problem (usually) is by looking to see if the datastore starts with "-iqn." or "_iqn.".
If it does, we assume it is not local storage; otherwise, it is assumed to be local storage (an incorrect assumption in this situation that leads to shared storage (that has already been added to CS as shared primary storage) being added to CS as local primary storage).

We need a more robust solution.
Does anyone know how to detect that a datastore does not reside on the local storage of the host in question?

Here is the relevant code (in HostMO.java):
public List<Pair<ManagedObjectReference, String>> getLocalDatastoreOnHost() throws Exception {
    List<Pair<ManagedObjectReference, String>> dsList = new ArrayList<Pair<ManagedObjectReference, String>>();

    ObjectContent[] ocs = getDatastorePropertiesOnHyperHost(new String[] {"name", "summary"});
    if (ocs != null) {
        for (ObjectContent oc : ocs) {
            DatastoreSummary dsSummary = (DatastoreSummary)VmwareHelper.getPropValue(oc, "summary");
            if (dsSummary.isMultipleHostAccess() == false && dsSummary.isAccessible() && dsSummary.getType().equalsIgnoreCase("vmfs")) {
                ManagedObjectReference morDs = oc.getObj();
                String name = (String)VmwareHelper.getPropValue(oc, "name");

                if (!name.startsWith("-iqn.") && !name.startsWith("_iqn.")) {
                    dsList.add(new Pair<ManagedObjectReference, String>(morDs, name));
                }
            }
        }
    }
    return dsList;
}

I see we already do this:
dsSummary.isMultipleHostAccess() == false
But that doesn't seem to work the way we thought.

I found an old e-mail from me on this:
<EMAIL_ADDRESS>


Hardening: Run an Ethereum long run test at baseline throughput for a week

Child of #316
Goal: Measure baseline throughput of common FireFly operations on top of Ethereum, at a steady load across multiple days. Flag anything that fails to deliver or takes unexpectedly long to confirm.

New Discord thread for runs in preparation for v1.1: https://discord.com/channels/905194001349627914/1004496575059476570

Run Report
Started: 8/17/22
Duration: ~92 hours
Git commit: d54355dc8d062930286a419140dceafb82d42903
Node Configuration
2 FireFly nodes on one virtual server (EC2 m4.xlarge)
Entire FireFly stack is local to the server (ie both blockchains, Postgres databases, etc)
Maximum time to confirm before considering failure = 1 minute

Configuration details
core-config.yml
log:
        level: debug
broadcast:
        batch:
                size: 200
                timeout: 1s
privatemessaging:
        batch:
                size: 200
                timeout: 1s
message:
        writer:
                count: 5
download:
        worker:
                count: 100
publicstorage:
                ipfs:
                        api:
                                requestTimeout: 2s
                        gateway:
                                requestTimeout: 2s

ethconnect.yml
rest:
  rest-gateway:
    maxTXWaitTime: 120
    maxInFlight: 200
    alwaysManageNonce: true
    attemptGapFill: true
    sendConcurrency: 3
    gasEstimationFactor: 2.0

instances.yml
stackJSONPath: /home/ubuntu/.firefly/stacks/v11rc1/stack.json

wsConfig:
  wsPath: /ws
  readBufferSize: 16000
  writeBufferSize: 16000
  initialDelay: 250ms
  maximumDelay: 30s
  initialConnectAttempts: 5
  heartbeatInterval: 5s

instances:
  - name: long-run
    tests: [{"name": "msg_broadcast", "workers":50},{"name": "msg_private", "workers":50},{"name": "blob_broadcast", "workers":30},{"name": "blob_private", "workers":30},{"name": "custom_ethereum_contract", "workers":20},{"name": "token_mint", "workers":10}]
    length: 500h
    sender: 0
    recipient: 1
    messageOptions:
      longMessage: false
    tokenOptions:
      tokenType: fungible
    contractOptions: {"address": "0xc39c9b8384d682a17b74d9454af3eeefff919434"}

Results

Broadcast messages: 3,899,387
Private messages: 5,077,650
Token mints: 381,309
Contract invocations: 2,090,955
No errors

Run terminated when disk space ran out and geth node terminated.
Run Graph


plot thumbnails to reflect config set when expanded

when ive minimized a visualization (i was looking at scatter) and gone back to where i can add another, its reasonable a user might be looking to have the configurations they set when expanded/full screen to still apply.

So if a user chooses log scale y axis in histogram, the next freshly made histogram should have log scale on by default?
Some settings (like log y scale) could be sticky, but others, like bin width, should not.
Currently, a user can copy/clone a visualisation if they wanted to preserve some settings.  I tried this with histogram.  You can set a custom log-scale and bin width, then clone the viz, then change the variable in the new viz, the log scale persists, but the bin width goes back to default, which is IMO the desired behaviour.

I mean when I configure a viz and then minimize it, the thumbnail version should probably keep the configuration. I didn't mean to apply the configuration to a new viz. Sorry I wasn't clear.

That's OK.
The behaviour you describe should be what happens!  There may be a bug.
Could you give more details please?
On Fri, Jun 11, 2021 at 11:23 AM Danielle Callan @.***>
wrote:

I mean when I configure a viz and then minimize it, the thumbnail version
should probably keep the configuration. I didn't mean to apply the
configuration to a new viz. Sorry I wasn't clear.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/VEuPathDB/web-eda/issues/172#issuecomment-859477798,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACLLHZ4WARFKV3CTNPPJCDTSHPZVANCNFSM46O5Y45A
.

Did a quick test. The scatter plot config (the plot type radio buttons) is
honoured in the thumbnail versions, so I don't see the problem you see -
yet!
On Fri, Jun 11, 2021 at 11:35 AM Bob MacCallum @.***> wrote:

That's OK.
The behaviour you describe should be what happens!  There may be a bug.
Could you give more details please?
On Fri, Jun 11, 2021 at 11:23 AM Danielle Callan @.***>
wrote:

I mean when I configure a viz and then minimize it, the thumbnail version
should probably keep the configuration. I didn't mean to apply the
configuration to a new viz. Sorry I wasn't clear.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/VEuPathDB/web-eda/issues/172#issuecomment-859477798,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACLLHZ4WARFKV3CTNPPJCDTSHPZVANCNFSM46O5Y45A
.

right. it may be that what im asking for isnt very easy. but the client side controls dont seem to persist. like if i click something off in the legend for ex. but im not sure a user has any reason to know why some things persist and others dont. itll probably be particularly noticeable for something like switching axes. because some types of plots i think client side switching makes sense, and others a new request should be made to the data service.

or maybe its even specific to the plotly legend, i havent played around w it too much yet tbh.

We are not persisting the result of legend interactions. I'm sure this is something we can address.

Oh I see.  Won't it be nasty persisting internal plotly things?  That's why we've disabled virtually all of the interactivity features.

implementation hint: plotly has a callback to capture entire state of a plot (ask @dmfalke for more)

post phase 1 - roll our own legend will make this work redundant (will need to redo it)

Tested in GEMS1 -> Bar plot of "Study timepoint" with "age group" as overlay. When I switch off an age group in legend and minimize the plot, when I open it again, the state persists (that age group is still off). Passes QA.

@nkittur-uga thank you for your tests! 👍


Watch doesnt receive motion notifications

Ive been trying to get the watch.py example to work with both my wemo switch and motion. The script is able to detect both of them, but when I toggle the switch, or move around I dont see the statechange receiver fire.
What should I be seeing on wireshark? I do see a HTTP SUBSCRIBE from my desktop to my wemo which are given a 200 OK by the wemo. I also see a few NOTIFY's coming out of the wemo but they dont  correspond with movement or switch activity.
This is the NOTIFY activity I see coming out of the switch. I dont see any directed at my desktop directly though.
NOTIFY * HTTP/1.1
HOST: <multicast-ip>:<multicast-port>
CACHE-CONTROL: max-age=86400
LOCATION: http://<wemo-switch-ip>:49154/setup.xml
OPT: "http://schemas.upnp.org/upnp/1/0/"; ns=01
01-NLS: <some long hash>
NT: urn:Belkin:service:manufacture:1
NTS: ssdp:alive
SERVER: Unspecified, UPnP/1.0, Unspecified
X-User-Agent: redsonic
USN: uuid:Socket-1_0-221241K0101C2C::urn:Belkin:service:manufacture:1

I have tried running this on my ubuntu desktop, and raspberry pi with the same result. I have a wemo switch and a wemo motion. Which I can control and check the status of without issue.

The root cause is that the server that is supposed to listen to UDP traffic is not started. I believe it should be the server from subscribe.py:95 I am not familiar enough with the project to know where to start this server.

ok so I found this commit (https://github.com/iancmcc/ouimeaux/commit/1aacda0d4b0a86ddd9b4cf9bf8e259e149b4459f) which fixed #63, but in the process it made it impossible to open the device discovery server and the notification server at the same time. This probably worked before caching was disabled?
Anyway, @iancmcc, would this change need to be reverted, or do I need two environments, one for discovery and one for notifications and they share devices somehow? Or is it possible to repurpose an environment for notifications after discovery is finished?


Adjust padding and header size on Profile layout

Natalie Hoffmann's name wraps on the profile page. Need to plan to accommodate longer names as headers.
Words should not split between two lines.
@pfulton - let me know if it would be helpful to mock up a profile with a much longer name for testing.

@pfulton - FYI, there is one viewport where this still breaks.

@pfulton - Can I also have you look at this layout with Jos again? The responsiveness leaves a lot of white space when viewed fullscreen on my laptop.


Casting results in wrong output type emitted in air

Currently equality is only present for primitive types it seems, to get around this limitation I decided to represent an enum as u64 and simply cast as a u64 when needed. Unfortunately this results in a panic as the wrong air type is emitted, in this case "Colour" was emitted.
#[derive(Eq, PartialEq, Copy, Clone)]
#[repr(u64)]
pub enum Colour {
    Red = 1,
    White = 2,
    Blue = 3
}

spec fn below(c: Colour, d: Colour) -> bool {
    let c = c as u64;
    let d = d as u64;
    let red = Colour::Red as u64;
    let blue = Colour::Blue as u64;

    c == red || c == d || d == blue
}

I got around this limitation by having external_body functions that handled the conversion for me, but thought that I should perhaps note this issue.

Hi!
To compare equality in spec code you can use equal(a, b) or (if you're using the new syntax) a === b.
Currently, as you've noted, our encoding doesn't handle the cast from an enum to an int. We could fix it to have a nicer error message though.

Thanks @tjhance, I shall use equal then, I shall close this issue in that case.


Merge reasonreact_jsx_ppx logic into the old one; fix a bug

Indicate we're using the old ppx through [@@@oldJSX].

<Foo.Bar /> now supported.

This is great!


BMP image loader fails to import images with 8, 4 and 1 bit depths

Godot version:
3.2
OS/device including version:
Windows 10
Issue description:
Try to load .bmp images with 8, 4 and 1 bit depth, it will fail to import them.
Loading 24 bit images works alright since #20038.
I know the bmp hate but here it is 🐛.
Minimal reproduction project:
bmp-8bit-import.zip

CC @SaracenOne

@SaracenOne note that I'm currently working on palette support in images that could help to solve this. Actually I've made some progress already on this, but you could fix it independently of whether the feature is merged or not (#28013).

For reference, the issue with importing 8-bit bmp images was fixed in my PR using the new palette API https://github.com/godotengine/godot/pull/28013/commits/45917aca3c9cc857ccbf25c3da48330fd4c0207a, I suppose it would be relatively easy to do the same with 4-bit and 1-bit images.


[UBS user-profile] server response 405 error to "/ubs/userProfile/user/save" http request and update name controller and method service.

Changed this method from save to update.
Fixed 405 bug.

Status 500


Add mainet checkpoints and update spork public key

I built a wallet from this code and it synched the entire blockchain without issues:
root@v9:~# artaxd
Artax server starting
root@v9:~# date; artax-cli getinfo | jq .blocks
Thu Aug  2 21:06:55 UTC 2018
3480
root@v9:~# date; artax-cli getinfo | jq .blocks
Thu Aug  2 21:13:48 UTC 2018
88487
root@v9:~# date; artax-cli getinfo | jq .blocks
Thu Aug  2 21:25:38 UTC 2018
185792
root@v9:~# artax-cli getblockhash 185792
a75129c87bf2923908af976c85156a4b736c9a2dea79567bd142cc739fba6208

Changing the spork public key as nobody seems to have the private key for the current one in the code.
The spork key is needed in order to activate a few sporks in the near future:
SPORK_10_MASTERNODE_PAY_UPDATED_NODES
SPORK_13_ENABLE_SUPERBLOCKS
SPORK_15_NEW_PROTOCOL_ENFORCEMENT_2

I rebased this against master and compiling a new linux build right now.
I'll post a comment when the build is ready for those that wants to test it.


Experience Replay memory

Just thinking about the memory size... Since you don't seem to reset the memory anywhere (sorry if I missed it), and it's a list, does that mean it's continuously growing during training from episode to episode? Could a sample experience from the memory (for a certain state) belong to episode x, as well as episode x+100?

yes, you are right. I didn't limit the memory size (though I should...). and a sample can be drawn from all past episodes uniformly. You are welcome to fix it or explore other sampling strategies!

ok thanks i'll look into it.


allow inference of output_shape from out array in affine_transform

Reference issue
closes #5258
What does this implement/fix?
Currently when output_shape is not provided output_shape is set to input.shape. This implies that the user must provide BOTH out and output_shape together whenever the shape of the output array's shape does not match the input.
It seems more intuitive to infer output_shape from the out array instead when it is provided.
As before, a RuntimeError is raised when both are provided, but the shapes do not match.
Additional information

Sounds like a good fix. CI is a bit unhappy.


Try to detect new dynamic heap usage at build time

Problem
We don't want additional heap use introduced unnecessarily or unknowingly.
Proposed Solution
It would be useful to have tooling that could statically detect calls that use heap.

I've looked previously and not found any existing tooling that can catch new heap calls. (One problem is that there is no definitive list of library calls that may use heap.)


Zarr Sprint Topics

Per our discussions in the bi-weekly GeoZarr SWG meeting, we identified a few focus tracks for the zarr sprint coming up on February 7/8th, 2024. In addition, I reviewed the original brainstorming ideas first discussed a year ago documented https://hackmd.io/t2DWpX1iQEWMKx1Fi4Px7A?both#Let’s-brainstorm. Many of these ideas are captured by the proposed list we discussed on January 24th. The topic of bidirectional interoperability with gdal is another clear theme, although as we discussed at the last SWG meeting, this would be very difficult to tackle in a single sprint and more importantly we may not have someone to lead this. Nevertheless I am listing it as an option to see if we could identify folks in the community to lead.
Here are the topics I have narrowed down to:

pyramiding @maxrjones, virtual
http browsable zarr @rabernat @kbgg
zarr v3 for zarr-python @jhamman, virtual
gdal bidirectional interoperability (???)

Here is the proposed template that I ask the folks who are tagged as leading the tracks above to complete and share below.
As A Zarr Sprint track...
Our focus is on <Outcome>
We believe it delivers <Impact> to <Whom>
This will be achieved when <some acceptance criteria>
The types of skills we need to complete this task are <some list>
We expect the level of difficulty to complete this to be <low, medium, high>

Topic leaders, if you can fill in the above template by Monday January 29th, then as a community, we provide ranked responses by Wednesday January 31st.

As a Zarr Sprint track focused on enabling support for V3 in Zarr-Python we are joining an ongoing effort working toward Zarr-Python version 3.0 (roadmap).
Our focus is on closing outstanding issues on the roadmap and testing the development branch in common geospatial applications.
Zarr-Python has traditionally been the canonical implementation of Zarr, therefore we believe this effort delivers immediate impact to the largest swatch of users, including those that use Zarr through downstream libraries (e.g. Xarray).
This will be achieved when any of the roadmap issues are closed or some of the following objectives are completed:

prototype using Zarr-Python 3's dev branch in Xarray's Zarr-backend
prototype using Zarr-Python 3's dev branch in Dask's Zarr IO utilities
benchmark typical geospatial read/write operations on groups/arrays/metadata to understand performance

The types of skills we need to complete this task are moderate to advanced familiarity with Python and Zarr.
We expect the level of difficulty to complete this to be medium to high.

A Zarr Sprint track focused on geospatial multi-scales / pyramids.
Our focus is on identifying and addressing shortcomings of the ndpyramid utility, either though development in that library or deciding where else development would need to happen.
ndpyramid is a utility for generating pyramids for Zarr datasets to enable performant visualization. The library was built specifically for use with the @carbonplan/maps toolkit and produces pyramids conforming to this schema. There has been persistent community interest in broader support and standards for pyramids in Zarr. The focus of this sprint is on establishing whether ndpyramid could provide the foundation for this support, and, if so, develop towards that goal, or, if not, establish where else development would happen.
We anticipate that this sprint will progress development towards geospatial pyramids in Zarr that can be used broadly for dynamic client visualization approaches, tiling servers, and multi-scale analysis. This will serve data providers, front-end developers, and researchers.
This will be achieved when:

We demonstrate the generation and usage of Zarr pyramids with @carbonplan/maps, titiler-xarray, QGIS, Datashader, and a SRCNN, all sharing the same schema.

Some potentially more attainable goals for the short sprint:

Bring your own data: Try out generating pyramids on your dataset and identify where current tools fall short
Define how different Tile Matrix Sets would be represented in the pyramid schema
Implement a proof of concept for producing a World CRS84 Quad pyramid using ndpyramid
Test a proof of concept for loading raw data and pyramids stored separately using a single datatree entrypoint

The types of skills needed complete this task are moderate familiarity with Python and Zarr. We would especially encourage participate from those familiar with geospatial projections, multiscale representations, and metadata conventions.
We expect the level of difficulty to complete this to be medium.

Will get this up tomorrow. Sorry for the delay.

Zarr Linked Hierarchy for HTTP-enabled Browsing
Focus and Outcomes
Our focus is on achieving the ability to explore nested Zarr groups over HTTP or other stores that do not provide a LIST-style operation.
This will enable

organizations to more easily share Zarr data on vanilla HTTP servers
web-based viewers for Zarr data to traverse nested groups

More Context

Zarr is not a file format; it is a specification for how to organize a nested hierarchy of numerical arrays and metadata in storage. In order to explore the contents of a Zarr hierarchy, clients generally need the ability to list the contents of directories in the storage layer. For filesystems or s3-compatible object storage, this is straightforward. However, most cloud-native geospatial data formats provide first-class read-only support via a vanilla HTTP protocol.
To address this need, Zarr V2 implemented a somewhat hacky “consolidated metadata” approach, in which all the metadata from a hierarchy are condensed into a single json file. This approach does not scale to very large, deeply nested Zarr hierarchies.
Now that Zarr V3 has been ratified, there is an opportunity to develop an extension that supports this HTTP-browsing use case in a more scalable and robust way. Specifically, we imagine developing a STAC-like mechanism for explicit links between parent and child groups that allow an HTTP client to quickly traverse a Zarr hierarchy.

Requirements

Zarr hierarchies of any size should be browseable and readable via HTTP without having to download the entire nested metadata
Clients entering a Zarr hierarchy should be able to traverse either up, until they reach the root group, or down, until they reach an array node.
Links can be generated and written by a client with list capabilities on the store. For example

A Zarr hierarchy on disk can be linked by traversing the directory store and adding the appropriate metadata for parent / child links; the entire hierarchy can then be served over an HTTP server.
A Zarr hierarchy on S3 can be linked by traversing the S3 store, adding the appropriate metadata for parent / child links; the entire hierarchy can then be accessed via standard HTTP GET requests (rather than S3 protocol).

Non-goals:

We do not aim to support writing to Zarr hierarchies via HTTP; this use case is read-only.
We assume the Zarr hierarchy exposed in this way is quasi-static; once the links are generated, they will not change frequently.

Implementation Plan and Skills Needed
We will try to implement this capability in zarr-python on the V3 branch. Contributors should be intermediate Python programmers (understand best practices around Python objects, typing, and code structure). Familiarity with the Zarr code base is not required but helpful. Participants should review the V3 roadmap and design document.
I'm also open to implementing this first in a javascript library, rather than Python. For example, in the source.coop viewers package.

Along the lines of my comment above, I have a concrete proposal that could be fun for someone (like @kylebarron 😉) to work on.
Zarr Python's V3 store interface is being redesigned to provide an all-async interface. The idea we have been discussing is to write a store on top of the Rust Object-Store crate. There are already Python bindings for this project but they are not async ready. If this particular plan is successful, it is possible this could become the core store in the zarr-python project.

@jhamman , been readying https://docs.rs/pyo3-asyncio/latest/pyo3_asyncio/index.html very carefully. Given I already did this once in rfsspec, I am prepared to give it a go on top of object-store. rfsspec showed marginal benefits, so while it may be worthwhile, do not expect a big return for the probably substantial effort.
Note that using rust async (tokio) in python async (asyncio) requires two event loops on two threads; it isn't simple! We also want to enable dask-style access from multiple (python) threads, so... Also, python bytes objects are annoying in rust (numpy buffers would be better, even for bytes output).

(I would be interested in this, because a rust-only zarr and kerchunk solution is very generally interested for those that need a C-level API; however, if we don't use numpy as the storage, and we don't have numcodecs directly, it maybe asks more questions than it answers; cf https://github.com/sci-rs/zarr ).

Note that using rust async (tokio) in python async (asyncio) requires two event loops on two threads; it isn't simple

FWIW the next version of pyo3 is likely to have big progress in async handling, and it sounds like it might no longer need two event loops? https://github.com/PyO3/pyo3/issues/1632#issuecomment-1752582018

Also, python bytes objects are annoying in rust (numpy buffers would be better, even for bytes output).

Why are they annoying? Is it because the memory is Python-allocated instead of Rust-allocated?

the memory is Python-allocated instead of Rust-allocated?

Yes, but also the internal immutability guarantee makes zero-copy handing the memory to/from rust hard. In rfsspec, I already wrote code around the python buffer protocol to cope with this, which appears to work but sidelines rust's memory protections.

An additional, v3 sprint topic idea, this one aimed at @TomNicholas. Manifest storage transformer. Specific goals for this sprint could be to:

Evaluate the proposal - https://github.com/zarr-developers/zarr-specs/issues/287
Hack together a small sample dataset using Kerchunk and bespoke translation code
Break ground on Zarr-Python's first v3 storage transformer - the logic is actually quite easy but the internal hooks are not complete.

In the Zarr pyramids breakout group, Thomas Maschler and I discussed the motivations for following the OGC TileMatrixSet 2.0 specification within the GeoZarr specification, which will be shared as a new issue to supersede https://github.com/zarr-developers/geozarr-spec/issues/30. We also discussed reading those TMS into rio-tiler using Xarray and started a refactor of ndpyramid to support the TMS specification.

Zarr-Python post-sprint update

@rabernat and @maxrjones worked on Zarr-Python's test environment and CI setup (https://github.com/zarr-developers/zarr-python/issues/1648):

https://github.com/zarr-developers/zarr-python/pull/1650
https://github.com/zarr-developers/zarr-python/pull/1655
https://github.com/zarr-developers/zarr-python/pull/1656

@d-v-b worked on removing the attrs dependency from Zarr-Python (https://github.com/zarr-developers/zarr-python/issues/1624)

https://github.com/zarr-developers/zarr-python/pull/1660

@kylebarron worked on a prototype store using new async Python bindings to Rust's object-store project

https://github.com/zarr-developers/zarr-python/pull/1661

Thanks all!

I added two example scripts for interactive geozarr in qgis.

attempt at a simple GeoZarr with overviews and the geozarr spec's metadata
add gdal supported _CRS attribute for crs detection in qgis

In the "chunk manifest / virtual concatenation" group our main outcome was a long technical discussion, which I've written up in ZEP-like form here https://github.com/zarr-developers/zarr-specs/issues/288#issuecomment-1939265240


How two checkpoints are selected in TransCoder?

For the TransCoder model, two separate checkpoints are provided for the following language directions.

TransCoder_model_1 for C++ -> Java, Java -> C++ and Java -> Python
TransCoder_model_2 for C++ -> Python, Python -> C++ and Python -> Java

Did you train TransCoder individually for the above two different directions setting? Or, the checkpoint is selected based on validation performances for those directions.

Hi. The checkpoints were selected based on the validation performances on a single run. We tried to select two checkpoints with close to the best validation performance for several pairs to avoid having to one model per direction.


[BUG] Error during alignment: RuntimeError: kaldi::KaldiFatalError

First of all, thank you for putting in so much work into this wonderful solution. It makes life easier being able to use Python to interface with Kaldi instead of having to string together commands all the time - this is nice!
Debugging checklist
[X --> v0.6.0] Have you updated to latest Kalpy version?
Describe the issue
During an attempt to make a simple function to align text to audio without any temp files (using GmmAligner.align_utterance), I encountered kaldi::KaldiFatalError during alignment, and couldn't trace the issue further.
For Reproducing your issue
All files involved in my testing are in the Drive folder here:

https://drive.google.com/drive/folders/148-z_-i_7U09QY8HgpUoeAE5_84U0zex?usp=sharing
The folder kalpy_testing/gh_raw_files is a copy paste from this repository's testing/data folder.
The Folder kalpy_testing/acoustic_model is the extracted acoustic model files for English (US) ARPA acoustic model v2.0.0
The Folder kalpy_testing/acoustic_model_mfa is the extracted acoustic model files for English MFA acoustic model v2.2.1
The test.py file is the python script I wrote, which you can run to reproduce the errors.

Extract the entire folder somewhere and make sure the file structure is the same as in the Drive folder.
Environment:
Run conda install -c conda-forge kalpy if you don't already have. (I used v0.6.0, since it is the latest version on conda-forge).
Run python test.py
This should reproduce the outputs:
Running custom forced alignment with no temp files...
Initializing lexicon...
Initializing aligner...
Initializing feature generator...
Performing forced alignment...
Computing features...
Compiling training graph...
Performing alignment...
Traceback (most recent call last):
  File "/app/test.py", line 242, in <module>
    align_one(input_wav, input_align_text, feature_generator, training_graph_compiler, aligner)
  File "/app/test.py", line 155, in align_one
    alignment = aligner.align_utterance(
  File "/env/lib/python3.10/site-packages/kalpy/gmm/align.py", line 72, in align_utterance
    ) = gmm_align_compiled(
RuntimeError: kaldi::KaldiFatalError

Additional context
I ran all of this in a GPU enabled Docker container on Ubuntu with Python v3.10.
I was ALMOST able to run the test in the repository test_align_sat_first_pass using the data in the kalpy_testing/gh_raw_files/kaldi folder corresponding to tests/data/kaldi in the repo (this is in the python file in the Drive folder I provided). The alignment was generated, but LexiconCompiler.phones_to_pronunciations failed to run, dumping the following error:
ERROR: SymbolStringToLabels: Symbol "" is not mapped to any integer label in symbol table <unspecified>
ERROR: Failed to compile string `           be          be                                       `, with token_type: symbol

I was NOT able to run the test in the repository test_align, and got the same RuntimeError: kaldi::KaldiFatalError as in my custom function.
I tried with two different dictionary+acoustic combinations:

English MFA dictionary v2.2.1 + English MFA acoustic model v2.2.1
English (US) ARPA dictionary v2.0.0 + English (US) ARPA acoustic model v2.0.0

Again, thank you for your work. I have used MFA for the past several months in projects, and am excited to get kalpy to work for even better interfacing with kaldi!

Okay, after playing around with it for a little while just now, I managed to set up a script that successfully performs forced alignment with no tmp files:
import os
import json

from kalpy.data import Segment
from kalpy.decoder.training_graphs import TrainingGraphCompiler
from kalpy.fstext.lexicon import LexiconCompiler
from kalpy.gmm.align import GmmAligner
from kalpy.feat.mfcc import MfccComputer

from praatio import textgrid as tgio

from _kalpy import feat, transform
from _kalpy.matrix import FloatMatrix
from kalpy.utils import read_kaldi_object

test_dir = "kalpy_testing"

def align_one(
    text: str = "we knew each other in school",
    wav_path: str = "16khz.wav",

    lexicon_compiler: LexiconCompiler | None = None,
    training_graph_compiler: TrainingGraphCompiler | None = None,

    lda_mat: FloatMatrix = None,
    feature_generator: MfccComputer | None = None,

    aligner: GmmAligner | None = None,
):
    # Compile training graph
    print("Compiling training graph...")
    training_graph = training_graph_compiler.compile_fst(text)

    # Get features
    print("Getting features...")
    splice_frames = 3
    feats = feature_generator.compute_mfccs_for_export(Segment(wav_path), compress=False)
    feats = feat.splice_frames(feats, splice_frames, splice_frames)
    feats = transform.apply_transform(feats, lda_mat)

    # Perform alignment
    print("Performing alignment...")
    alignment = aligner.align_utterance(training_graph, feats)

    # Generate intervals
    print("Generating intervals...")
    intervals = alignment.generate_ctm(aligner.transition_model, lexicon_compiler.phone_table, frame_shift = 0.01)

    # Get actual words
    print("Getting actual words...")
    actual_words = [lexicon_compiler.word_table.find(x) for x in alignment.words]
    text = " ".join(actual_words)
    print(text)

    # Convert phones to pronunciations
    print("Converting phones to pronunciations...")
    ctm = lexicon_compiler.phones_to_pronunciations(text, alignment.words, intervals)

    # Get textgrid tiers
    print("Getting textgrid tiers...")
    file_duration = 4.07
    word_tier = tgio.IntervalTier("words", [], minT=0.0, maxT=file_duration)
    for w in ctm.word_intervals:
        word_tier.insertEntry(w.to_tg_interval(file_duration))

    return word_tier

if __name__ == '__main__':
    # Kalpy testing to replace MFA for forced alignment
    # https://github.com/mmcauliffe/kalpy/blob/main/tests/test_align.py

    print("Running custom forced alignment with no temp files...")
    align_options = {
        "transition_scale": 1.0,
        "acoustic_scale": 0.1,
        "self_loop_scale": 0.1,
        "beam": 1000,
        "retry_beam": 40,
    }

    # File paths
    input_wav = '16khz.wav'
    input_align_text = 'we knew each other in school'

    # acoustic_model_dir = f"{test_dir}/acoustic_model"
    acoustic_model_dir = f"{test_dir}/acoustic_model_mfa"

    acoustic_model_base_path = f"{acoustic_model_dir}/final.mdl"
    acoustic_model_tree_path = f"{acoustic_model_dir}/tree"
    acoustic_model_alignment_path = f"{acoustic_model_dir}/final.alimdl"
    acoustic_lda_mat_path = f"{acoustic_model_dir}/lda.mat"

    # dictionary_path = f"{test_dir}/english_us_arpa.dict"
    dictionary_path = f"{test_dir}/english_mfa.dict"
    lexicon_path = f"{test_dir}/lexicon_fsts.ark"

    model_metadata_path = f"{acoustic_model_dir}/meta.json"

    # Get model metadata
    metadata = json.load(open(model_metadata_path, 'r'))
    phones = metadata['phones']
    features_metadata = metadata['features']

    snip_edges: bool = features_metadata['snip_edges']
    use_energy: bool = features_metadata['use_energy']

    # Initialize Lexicon
    print("Initializing lexicon...")
    lexicon_compiler = LexiconCompiler(position_dependent_phones=False, phones=phones)
    lexicon_compiler.load_pronunciations(dictionary_path)

    if not os.path.exists(lexicon_path):
        print("Creating lexicon FSTs")
        lexicon_compiler.fst.write(lexicon_path)

    training_graph_compiler = TrainingGraphCompiler(acoustic_model_base_path,
                                                    acoustic_model_tree_path,
                                                    lexicon_path,
                                                    lexicon_compiler.word_table)

    # Initialize Aligner
    print("Initializing aligner...")
    aligner = GmmAligner(acoustic_model_alignment_path, **align_options)
    aligner.boost_silence(1.0, lexicon_compiler.silence_symbols)

    # Initialize feature generator
    print("Initializing feature generator...")
    feature_generator = MfccComputer(snip_edges=False)

    # Initialize LDA matrix
    print("Initializing LDA matrix...")
    lda_mat: FloatMatrix = read_kaldi_object(FloatMatrix, acoustic_lda_mat_path)

    # Perform forced alignment.
    print("Aligning one...")
    word_tier = align_one(
        text=input_align_text,
        wav_path=input_wav,
        lexicon_compiler=lexicon_compiler,
        training_graph_compiler=training_graph_compiler,
        aligner=aligner,
        feature_generator=feature_generator,
        lda_mat=lda_mat
    )

    for (interval) in word_tier.entries:
        print(interval)

    print("Forced alignment completed!")

Here are some things that caused it to break along the way:

If lda_mat = None, it didn't work. So I extracted the part that applies lda transformation from the FeatureArchive class and applied it to the extracted features right before performing alignment
Using ARPA dictionary + ARPA based acoustic model didn't work. I got it to work with English MFA dictionary v2.2.1 + English MFA acoustic model v2.2.1

It takes a while to run LexiconCompiler.phones_to_pronunciations after alignment when I want to get the words and their timestamps, but it is very precise. Is there a faster way to obtain the IntervalTier for just the words?
Awesome library, thank you for your work!

Update --> I made a mistake which resulted in ARPA not working due to the check if not os.path.exists(lexicon_path). I removed the file check and let it overwrite every time, and it works with ARPA as well now.
ARPA runs significantly faster. About 0.8s to process a 4s audio file, instead of 16s when using the MFA dict.
I'm going to close this - I just had to learn how to use the library properly!
Have a nice day :-)


Update Wanwan Aijou Monogatari.md

Recommendations for updates to the translations. Feel free to push back/reject if I'm in the wrong on any of these. :)

I thought "Barkley" would be a closer translation for the original バク, which seems to be a play on the word "bark."
The teddy bear's name in the original is ムク (Muku), which I'm not certain is a rearrangement of the word "bear" くま (kuma). I would propose keeping the name as Muku for now.
Replaced some instances of kawaii with cute.
Fixed a typo, assorted miscellany.

Oh hey! I have no idea why but I didn't get a notification for this. Thank you! I should have fixed the typos from this PR now, but give me a little time to mull over the changes to Muku/Earb and Barkley/Barley :)

I agree that Muku's name is not an exact mirror of kuma, but here's why I think it was either intended that way or at minimum it works that way:

So many of the characters' names are very simple reflections of what they are (Clock, Cabbage, Pumpkin, Gunbot, etc.), so I think muku the kuma is one of those.
While it's not an exact mirror, I think it still rolls off the tongue "muku kuma" "kuma muku"
If I read "maku" my mind goes to "Mac" in English, so maybe they wanted to tweak it to avoid this? Also maybe they preferred "muka" since "muka" is also a funny exclamation, like I'd translate it "GRAH!" or "GUH!!!" or "GAH!" OK, this doesn't explain why it's a mirror of "kuma" but it might help explain why it's not a perfect mirror of "kuma".
But my favourite reason that I think ムク is a mirror of クマ, is that mu/ma ム/マ is a mirrored letter in itself, and Momo was a little kid when she named him, so I think she might have written the letter "backwards" unintentionally and it stuck. Just like the familiar stereotype in the west of little kids writing one or two letters backwards.

I'm still going to see if I can ask a native speaker what they think, but I wondered if hearing this would change your mind :)

I see... interesting! I am intrigued by the backward-letter theory!
For me, the name Earb stuck out because it took me a while to understand how to read or say it (Eerb?) or realize its connection to Bear. The current spelling most reminds me of Pig Latin (Earbay? Airbay?), which is why I was assuming it was a kuma/maku thing. As for idea of matching a kid's misspelling/miswriting, that's tough. The English equivalents might be "Bare," "Bar," "Ber," "Bair," etc.
All this to say... maybe I don't have a good recommendation at all! Haha. Would love to hear thoughts from native Japanese speakers for sure. If we knew the pun for sure, it would help a ton.


Revise ParseKdlNode trait to return Result instead of Option

Currently the ParseKdlNode trait that we use to parse policy files returns Option<T>, making it impossible to report/handle errors during parsing other than logging the error and returning None.

Not our fault, ParseKdlNode is from another crate.


vcpkg exists immediately if /vcpkg/config can not be found

Since 67ce764c vcpkg does terminate immediately if the config file in AppLocal can not be found.
The reason for this is that in https://github.com/Microsoft/vcpkg/blob/master/toolsrc/src/vcpkg.cpp#L114 the .get_or_throw() changed to .get_or_exit() but the code at that location relies on an exception being thrown on error (it's located in an try-catch-block).
I think reverting the line back to .get_or_throw() should fix the issue. I did not check whether there are other locations in the code that suffer from the same issue.

Sorry, just yet realized that this is the same issue reported in #866.


cannot create project

Error while creating release definition: The classic pipelines are disabled for this project / organization.

Go to organizational setting -> policies -> enable allow third-party access. Once you do the below settings you can try provisioning the project again.


Deleting element from string set with mock_dynamodb2 doesn't match AWS

Versions: python==3.8.5, moto[iam]==1.3.15
When deleting the final element from a string set attribute in DynamoDB on AWS via update_item, the attribute is dropped. This behaviour is not specified in the AWS docs, but is observed in production.
In moto however, the string set attribute remains present but without any entries. Therefore moto is not matching production AWS behaviour. It is certainly a question, given this is undocumented behaviour, whether moto should change to match.
Consider the following code:
import boto3
import moto
import os
import pytest

TABLE_NAME = 'table'

@pytest.fixture(scope='function')
def aws_credentials():
	"""Mocked AWS Credentials for moto."""
	os.environ['AWS_ACCESS_KEY_ID'] = 'testing'
	os.environ['AWS_SECRET_ACCESS_KEY'] = 'testing'
	os.environ['AWS_SECURITY_TOKEN'] = 'testing'
	os.environ['AWS_SESSION_TOKEN'] = 'testing'

@pytest.fixture(scope='function')
def dynamodb(aws_credentials):
	with moto.mock_dynamodb2():
		client = boto3.client('dynamodb', region_name='eu-west-1')
		client.create_table(
			TableName=TABLE_NAME,
			KeySchema=[
				{
					'AttributeName': 'customer',
					'KeyType': 'HASH',
				},
			],
			AttributeDefinitions=[
				{
					'AttributeName': 'customer',
					'AttributeType': 'S',
				},
			],
		)
		yield client

def test_conidtional_delete(dynamodb):
	customer_name = "Joe Bloggs"
	order = "An Order"
	dynamodb.update_item(
		TableName=TABLE_NAME,
		Key={'customer': {"S": customer_name}},
		UpdateExpression=f"ADD orders :order",
		ExpressionAttributeValues={
			':order': {"SS": [order]},
		},
	)

	response = dynamodb.scan(
		TableName=TABLE_NAME,
		ProjectionExpression='customer, orders',
	)

	print(f"Response 1: {response}")

	dynamodb.update_item(
		TableName=TABLE_NAME,
		Key={'customer': {"S": customer_name}},
		UpdateExpression=f"DELETE orders :order",
		ExpressionAttributeValues={
			':order': {"SS": [order]},
		},
	)

	response = dynamodb.scan(
		TableName=TABLE_NAME,
		ProjectionExpression='customer, orders',
	)

	print(f"Response 2: {response}")

Which outputs the following:
Response 1: {'Items': [{'customer': {'S': 'Joe Bloggs'}, 'orders': {'SS': ['An Order']}}], 'Count': 1, 'ScannedCount': 1, ...}
Response 2: {'Items': [{'customer': {'S': 'Joe Bloggs'}, 'orders': {'SS': []}}], 'Count': 1, 'ScannedCount': 1, ...}

When executed against production AWS however, the responses are:
Response 1: {'Items': [{'customer': {'S': 'Joe Bloggs'}, 'orders': {'SS': ['An Order']}}], 'Count': 1, 'ScannedCount': 1, ...}
Response 2: {'Items': [{'customer': {'S': 'Joe Bloggs'}}], 'Count': 1, 'ScannedCount': 1, ...}

Thanks for raising this @jamesgunnfiveai - will mark this as an enhancement.


Fixes issue #183

Actually this is a workaround, not a fix because the bug is in closure compiler. I'll break it down in detail below.
First understand that the function axs.properties.findTextAlternatives can legitimately return null.
Then take a look at the way we call it:
var textAlternatives = axs.properties.findTextAlternatives(element, {});
if (!textAlternatives || textAlternatives.trim() === '')

All good. But then take a look at what closure compiler produces:
return "" === axs.properties.findTextAlternatives(a, {}).trim() ? !1 : !0;

That will throw an exception if the return value is null.
The easiest workaround is to get closure compiler to produce something else. The proposed change does not change the logic of the audit but results in safe code generated by closure compiler:
a = axs.properties.findTextAlternatives(a, {});
  return null == a || "" === a.trim() ? !1 : !0;

@alice PTAL

Please update the Changelog: https://github.com/GoogleChrome/accessibility-developer-tools/blob/master/Changelog.md - manual process to help me write release notes. Sorry for the overhead :(
Once that's done and the nit addressed, this is :+1: - definitely an improvement regardless of the closure weirdness.

No worries about the changelog overhead, it is well worth the effort IMHO.


The $includePaths in EncodingParameters does not seem to be honored

In the showSparseAndFieldSetsExample() method of the sample application on line 88 the $includePaths parameter is declared as ['posts.authors'] but only the site resources is shown.
Changing the parameter to [] displays the full document as expected, but changing this to any value simply includes the site resource.
One would expect ['posts'] to include the posts resource as a related resource, but it does not.

Actually I have very similar test which appears to be OK. I'll look closer what's the problem with sample app.

That's a bug in a sample app. As fieldset is actually attributes and relationships the proper EncodingParameters should be
        $options = new EncodingParameters([
            // Paths to be included. Note 'posts.comments' will not be shown.
            'posts.author'
        ], [
            // Attributes and relationships that should be shown
            'sites'  => ['name', 'posts'],
            'posts'  => ['author'],
            'people' => ['first_name'],
        ]);

I'll update the sample and close the issue.

Thanks for the clarification. I played around with that a bit and it seems to work.
However, just a comment on this behavior: As someone new to the library, this way of specifying/configuring the include parameter does not feel intuitive.
I would expect to be able to specify the include paths and field sets separately. Mixing in the links with the fields array seems messy. Not sure how else to do it though?
Maybe a nested array?
$options  = new EncodingParameters(
    ['posts.author'],
    [
        'sites'  => [
            'fields' => ['name'],
            'links' => ['posts']
        ],
        'posts' => ['links' => ['author']],
        'people' => ['fields' => ['first_name']]
    ]
);

Breaking change
It won't be able to distinguish fields from links as a result it's just a matter of time for a bug to be posted

Not sure I follow that. But I am sure there is a technical reason for it.
Thanks for looking into it.

Will be closed when merged to master. Issue locked.


Pass through kwargs in get and post requests

Tests are needed to demonstrate what's fixed and to prevent regressions.

Closing due to inactivity.


__decorate is not defined error

After the update to 0.7.8 i have started getting the following errors in my console
Uncaught ReferenceError: __decorate is not defined
    at Object.exports.GLOBAL_INPUTS.BOOTSTRAP (config.js:30)
    at __webpack_require__ (bootstrap 6672543…:52)
    at Object.exports.isArray (index.js:2)
    at __webpack_require__ (bootstrap 6672543…:52)
    at Object.<anonymous> (index.js:23)
    at __webpack_require__ (bootstrap 6672543…:52)
    at Object.678 (baseapi.service.ts:116)
    at __webpack_require__ (bootstrap 6672543…:52)
    at Object.797 (app.component.ts:125)
    at __webpack_require__ (bootstrap 6672543…:52)
    at Object.673 (src async:34)
    at __webpack_require__ (bootstrap 6672543…:52)
    at Object.1310 (main.bundle.js:84)
    at __webpack_require__ (bootstrap 6672543…:52)
    at webpackJsonpCallback (bootstrap 6672543…:23)

I am using angular-cli 1.0.0-beta.30 with node 7.5.0 on macOS
Edit: Changed version to 0.7.8

In 0.7.8 also happens? : )
Thanks hassanasad !

oh sorry - i meant to say it started happening in 0.7.8 (not in 0.7.7) :)

Fixed, update to ng2-responsive : 0.8.3.
Thanks, @hassanasad

Thanks @ManuCutillas - everything is now back to normal again :)


remove submodules and use native ansible-y bits

original pr was here https://github.com/rcbops/rpc-openstack/pull/578

the 'default' test needs to be retargeted from master to liberty I don't have perms to do that.