We need to choose one WYSIWYG and remove the other, discuss!

@frjo, @todd-dembrey, @fourthletter @Parbhat
What are the pros/cons. One uses markdown the other HTML. One allows pasting in rich text (the html one), the other doesn't. Others? I really want us to come to a conclusion, noting that neither are prefect.

My option would be to keep the wysiwyg editor as is for the public pages and use markdown for everything else.
There already exist a markdown textfield for applications forms, ready to use. Same implementation is also used for submission comments today. The editor in use is ok but not perfect. I would like to find a better one but that can be done without it effecting the content in any way, just makes it nicer for the users.
The third place is the nodes editor in the react app. Hopefully it would not be too hard to switch that out for markdown.

I am new to the rich text editors discussions/choices used in this project. Looks like we use TinyMCE for the apply site and pagedown (markdown editor by StackOverflow) mostly in public site and apply site's MarkdownTextFieldBlock.
Pagedown is used mostly in the introduction field of Wagtail admin pages. The new rich text editor Draftail came in Wagtail 2.0 support markdown and new features are being added to it. It seems better to streamline it across the wagtail admin. But will require extra work in migrating content from pagedown markdown to HTML compatible with Wagtail's rich text editor.

Since Draftmail is built in to Wagtail it would be good to replace TineMCE with it. It should most likely be used for the introduction fields as well, I added markdown to get formatting working quickly.
There is also a React editor for notes in the react app for staff rounds administration, https://github.com/sstur/react-rte. Do not know how easy or important it's to replace that. (I would be happier if we could replace react completely with something simpler, this project is complex enough as it's.)


[full api] e-entity/dbpedia-spotlight/documents fails

dbpedia-spotlight request with default values exits with the following error message on both freme-live and freme-dev:
{
  "exception": "eu.freme.common.exception.ExternalServiceFailedException",
  "path": "/e-entity/dbpedia-spotlight/documents",
  "error": "Bad Gateway",
  "status": 502,
  "timestamp":<PHONE_NUMBER>064
}

The error message indicates that DBPedia spotlight is not reachable. This is not a bug on the FREME side but on the DBPedia spotlight side. I am sure that the endpoint will be reachable again in the future but we have no influence on this.


Routing error when deploying Swarm templates

Rancher Version: 1.0.0-rc2
Docker Version: 1.10.3
OS: Boot2Docker version 1.10.3
Steps to Reproduce:

Create a Swarm environment
Navigate to catalog, and select details for any entry
Select a version and click launch

Results: The user is routed to the swarm services page
Expected: The user is routed to the swarm projects page

Version - v1.0.0-rc4
Verified fixed


HookedServiceEventDispatchers should provide an asStream() method

And then one for multiple methods (chain them together):
app.service('foo').beforeCreated.asStream().listen(...);
app.service('bar').beforeStream(['indexed', 'created']).listen(...);


[Bug]: messageId not listed as valid parameter to .track()

Where on segment.com/docs did you see this issue?
https://segment.com/docs/connections/sources/catalog/libraries/server/node/#track
What browsers are you seeing the problem on?
Chrome
Please describe the issue.
The documentation does not mention that you can send messageId (from JS, I guess it is named message_id in your other clients), even though that is what you have to send for handling duplicate data.
Looking at the source code for your JS client it explicitly uses the messageId passed in if there is one, or generates one if there is none.

@stayseesong Can you take a look at this one when you have a moment?


Fix: Enable open sidebar filter links in a new tab

Fixes #2139

Earlier it was not possible to open the sidebar filter links as real links in a new tab
Enclose the links inside an anchor tag to get the out of the box functionality of open in new tab

Before the change

After the change

@iamareebjamal @mariobehling Can you please review?

And the normal clicking and filtering is working both in this page and when opening in a new tab?

@iamareebjamal yes it is working fine.


Fix ApnsPHP_Push to not send a pcnt_signal_dispatch if is not in serverMode

If running standalone ApnsPHP_Push in a custom cli script that uses pcntl signals, sending APNS notifications will affect the proper function of it.

It makes sense, but can you explain what's the problem with pcntl_signal_dispatch in your custom script? Also consider this change might break existing code that relies on that dispatch.

For what I understand from the code the pcntl_signal_dispatch is used in ApnsPHP_Push::send method because is necessary when it is used by the ApnsPHP_Push_Server subclass.
Checking if the function exists is not the best approach to execute pcntl_signal_dispatch since you might be using ApnsPHP_Push in a CLI script that uses pcntl signals for other purposes.
People shouldn't be relying in pcntl signals when using ApnsPHP_Push since that is a standalone class to send APNS notifications which requirements are: if you plan to use only Push and Feedback provider without the Server part you need only OpenSSL (no PCNTL, System V shared memory or semaphore)

@jbaez ok, but can you explain your specific problem?

Yes, we have a script that uses pcntl_fork to create child processes which run beanstalk jobs among other things.  Thus pcntl_signal_dispatch would make the child quit?

@lethalman the worker CLI script is running as a "daemon" doing a pcntl_signal_dispatch in a loop.
Can you debug more the situation? I think pcntl_signal_dispatch is actually catching a problem in your app.

Well I debugged it, the problem was in the child job that involved sending APNS messages which uses this library, and pcntl_signal_dispatch was causing the problem. The bug doesn't occur all the time, normally I have to leave it running for a day to occur which makes it very difficult to debug. When it happens beanstalk (we are sending APNS messages using beanstalk) enters in an endless loop trying to send 2 APNS messages. They never finish as the SIGCHLD is killing the job before it's finished.
This fix in the library solved the problem.

@jbaez it's not a "fix", it's a fix for your program that you applied to this library. I'm sure existing applications are relying on apns triggering signals in the loop.
I could accept a different patch: add an option to disable the dispatch. That is, by default it's on also for non-server mode, but the user can choose to disable it, though I don't see any reason for disabling it.

@lethalman It is ok with me you don't want to accept the patch, but in my opinion there is no need to use pcntl_signal_dispatch in this class. I don't think people using this class are expecting it to send pcntl_signal_dispatch signals (which is also not documented) and it's only intention is to work with ApnsPHP_Push_Server subclass.
If someone using the library would like ApnsPHP_Push to send pcntl_signal_dispatch for any reason they just have to subclass it and in the constructor set $this->_bServerMode = true;
Anyway, thanks for getting involved and try to suggest fixes, as you said a simpler way of avoid it sending signals is to add a public setter in ApnsPHP_Push class to modify the serverMode flag, I just chose not to because I don't see the reason why it should be sending signals.

@jbaez the reason for dispatching signals is that's a busy loop, and it would block signals if you don't dispatch them. It's quite natural for scripts, whether you are using the server class or your own customized server class. So the dispatch must be there in any normal case.
Subclassing just to set a flag is inconvenient, I'd rather have an explicit setDispatchSignals.

@lethalman Ok then, I will add a commit to the pull request using the setter instead when I get some free time.


Iter

Implement RFC for Additional Iter methods, resolving #1370

As part of the RFC, we discussed adding top level documentation. Can you do that before we merge?
The Iter documentation is really scant. Listing the class doesn't give any real indication of how to use this package. PonyTest's top level docs are a good baseline for what we should have.

The examples in the package level feel more like docs on methods. What the package level should have is "what is this? how do I used it? how do I put these things together?"
From the example(s) you gave when we were discussing for the RFC, a lot of the power comes from stringing these methods together. How does that work? How does run play into that? How do I get final results? When are these lazy, when are they greedy? A kind of "So you want to use Iter eh? Here's how!" documentation.

@Theodus those changes are excellent.
so how do i know what is lazy and what isn't?
that could be at the package level of listing them
in could be method naming convention
it could be in method doc-strings.
however i learn what is lazy and what isn't, that should also be in the package level docstrings.

@SeanTAllen lazy methods return Iter types. Should I just say that in the package level docstring?

CHANGELOG has been versioned. You'll need to rebase against master @Theodus.

@Theodus yeah, I think that would be good. Gives you a broad overview of the landscape before digging.

@Theodus
This looks like an excellent start for docs with this. Hopefully they get fleshed out more with time but given the lack of people chipping in on docs, that seems unlikely so, the effort now is worth it.
You want to squash and we can merge it?

@SeanTAllen squashed

Done!
Thanks @Theodus.


fix(shell): maximize respect height of active screen

this pr introduces the prerelease of https://github.com/microsoft/CsWin32 for sourcegenerating pinvoke commands.
the methods that are to be pinvoked are defined in the NativeMethods.txt file in the gitout folder.
we need to be willing to use this and also be aware of for instance this thread if we want to merge this pr

resolves go-3

Close in favor of https://gitout.myjetbrains.com/youtrack/issue/go-85


Add PubSub publish timeout

This adds a timeout to publishing messages to PubSub, as well as
improves some logging in the default retry implementation.
The initial timeout implementation here is done a semver-compatible way,
and with a relatively simple configurable fixed timeout. In the next
major semver I'd like to make the configuration required, and
potentially add some time-based retrying and timeout backoff.

In the next major semver I'd like to make the configuration required, and potentially add some time-based retrying and timeout backoff.

I see logic for exponential backoff and that code is part of the pubsub client struct. Are we just not using that logic yet, or is this referring to something else?

In the next major semver I'd like to make the configuration required, and potentially add some time-based retrying and timeout backoff.

I see logic for exponential backoff and that code is part of the pubsub client struct. Are we just not using that logic yet, or is this referring to something else?

There's exponential backoff for retries (i.e. try again after increasing sleep). Timeouts could also use such a backoff (i.e. wait for a response with increasing timeout), instead of the current fixed timeout, and potentially the same or similar backoff code. However the current retry logic is count-limited and not time-limited, which means if the backoff is expontential and the timeout is exponential, retrying a series of timed-out requests will take N attempts which might add to many minutes of time. If retries were time-limited, then there'd be a better overall cap to the possible delay from a series of timeouts.
Having time-limited retries is a worthwhile change, but will take some work and I want to get these timeouts in place first.


--split_events not returning new .nc file of events

I'm trying to run a simple script to get the flash extent density for some GLM files and I'm running into some bizarre behavior.
I've installed the glm via https://github.com/deeplycloudy/glmtools/blob/master/docs/index.rst with no installation errors, but when I try running the  glm_test_data_new_grid_dev.ipynb I can't get the separate event .nc files that are needed.
From the example notebook:
import subprocess
import os, glob
import tempfile
tmpdir = tempfile.TemporaryDirectory()

import glmtools
from glmtools.test.common import get_sample_data_path
# glmtools_path = os.path.abspath(glmtools.__path__[0])
glmtools_path = os.path.abspath('.')

# Set the start time and duration
startdate = datetime(2018, 7, 2, 4, 30)
duration = timedelta(0, 60*5)
enddate = startdate+duration

cmd = "python {0}/grid/make_GLM_grids.py -o {1}"
cmd += " --fixed_grid --split_events --goes_position=east --goes_sector=meso"
cmd += " --ctr_lat=33.5 --ctr_lon=-101.5 --dx=2.0 --dy=2.0"
cmd += " --start={3} --end={4} {2}"

cmd = cmd.format(glmtools_path, tmpdir.name, ' '.join(samples),
                startdate.isoformat(), enddate.isoformat())
print (cmd)
#out_bytes = subprocess.check_output(cmd.split())
# print(out_bytes.decode('utf-8'))
print()
grid_dir_base = tmpdir.name
nc_files = glob.glob(os.path.join(grid_dir_base, startdate.strftime('%Y/%b/%d'),'*.nc'))
print(nc_files)

I get back an empty nc_files list:
python /glmtools/examples/grid/make_GLM_grids.py -o

/var/folders/c2/404k4rn50k53k7nrlzmsh3br0000gn/T/tmpg4lk6ydh --fixed_grid --split_events -

-goes_position=east --goes_sector=meso --ctr_lat=33.5 --ctr_lon=-101.5 --dx=2.0 --dy=2.0 --

start=2018-07-02T04:30:00 --end=2018-07-02T04:35:00

/glmtools/glmtools/test/data/OR_GLM-L2-LCFA_G16_s20181830433000_e20181830433200_c20181830433231.nc

/glmtools/glmtools/test/data/OR_GLM-L2-LCFA_G16_s20181830433200_e20181830433400_c20181830433424.nc

/glmtools/glmtools/test/data/OR_GLM-L2-LCFA_G16_s20181830433400_e20181830434000_c20181830434029.nc

[]

I'm expecting to see several .nc files such as:
...flash_init.nc
...flash_extent.nc
...source.nc
...total_energy.nc
...footprint.nc
...group_init.nc
...group_extent.nc
...group_area.nc
...flash_area_min.nc
Any ideas on what I'm doing wrong? I can provide cat make_GLM_grid.log details if that helps.

Hi @MethaneRain, to take your last question first, the master branch now outputs a unified grid file format, where all grids files are in the same .nc file.
We also changed how the output path works. Instead of creating a default path convention, it is required to specify the exact output path you want. The new_grid_dev notebook is outdated. Try using plot_glm_test_data.ipynb instead, which should be current. I suspect the code ran and produced files, but the glob command that went to look for files was no longer looking in the correct spot.

This worked, thanks for the heads up on the update, much appreciated!


Error when opening Adobe XD Exported Menus.

Expected results: Screen change to Main Screen
Actual results: I/flutter ( 5763): unhandled element filter; Picture key: StringPicture(String#19741, colorFilter: null)

Hi @origamisimplicity,
When you say Adobe XD, are you referring and using this plugin ?

Hi @origamisimplicity,
Please provide flutter doctor -v and a minimal complete reproducible code sample that shows the issue you are facing.
Thanks.

[√] Flutter (Channel stable, v1.17.5, on Microsoft Windows [Version 10.0.19041.329], locale pt-PT)
• Flutter version 1.17.5 at C:\flutter
• Framework revision 8af6b2f038 (3 weeks ago), 2020-06-30 12:53:55 -0700
• Engine revision ee76268252
• Dart version 2.8.4
[√] Android toolchain - develop for Android devices (Android SDK version 30.0.1)
• Android SDK at C:\Users\nunom\AppData\Local\Android\sdk
• Platform android-30, build-tools 30.0.1
• Java binary at: C:\Program Files\Android\Android Studio\jre\bin\java
• Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b01)
• All Android licenses accepted.
[√] Android Studio (version 4.0)
• Android Studio at C:\Program Files\Android\Android Studio
• Flutter plugin version 47.1.2
• Dart plugin version 193.7361
• Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b01)
[√] VS Code (version 1.47.2)
• VS Code at C:\Users\nunom\AppData\Local\Programs\Microsoft VS Code
• Flutter extension version 3.12.2
[√] Connected device (1 available)
• Android SDK built for x86 • emulator-5554 • android-x86 • Android 8.1.0 (API 27) (emulator)
• No issues found!

I have a simple button calling the menu that i exported from Adobe XD.
 if (numid.isNotEmpty) {
        print("valid");
        Navigator.push(context, MaterialPageRoute(builder: (context) => MainScreen()));
      } else {
        print("invalid");

Hi @origamisimplicity,
That sounds like an issue with the plugin you are using. Since Adobe XD is a 3rd party plugin, I'd suggest you to log this issue in their repo (https://github.com/AdobeXD/xd-to-flutter-plugin/issues).
I am going ahead and closing this since it doesn't seem to be an issue with Flutter. If you disagree, write in comments and I'll reopen.
Thanks.


Added support for specifying a precision with which to compare elements of float arrays (fixes #505)

Good work ! Integrated.

Thanks!
It was just copy&paste of your work on doubles.

You got me, I was really praising myself ! :smile:

:smile:


Sidebar Menu is updated in rapidly after hot reload

Bug Report
Describe the bug
The Sidebar is updated rapidly, after hot reload.
When left alone for a while,  the browser tab gets slugish.
The links are not Clickable.
After a full reload everything is fine.
To Reproduce
Repo

git clone git@github.com:theparthy/docz-sidebar-issue.git
Run npm run docz dev
got to src/elements/Spinner/index.mdx
remove or add Component in Playground
After hot reload look at sidebar in Elements tab in DevTools. (Screenshot Below)

Expected behavior
Only one update of Sidebar
Environment

docz version: [2.1.1]
OS: [OSX 10.14.6]
Node/npm version: [v12.3.1]
Other libraries used

    "reakit": "^1.0.0-beta.13",
    "styled-components": "^4.4.1",
    "styled-icons": "^9.0.1",
    "styled-is": "^1.3.0",
    "styled-system": "^5.1.4"

Additional context/Screenshots

Hey @theparthy
Thanks for catching and reporting that bug, it's an annoying one !
The fix will be released with 2.2.0.
Meanwhile you can use 2.2.0-alpha.7 (yarn add docz@next) to use the patched code right away 👍

Hey @rakannimer
thanks for the quick response and fix.
2.2.0-alpha.7 fixes the issue.

Awesome ! You're welcome !
Going to close this, feel free to re-open or comment here if it re-occurs 👍


llama support for the generation of the code

I can't pay for the ChatGPT 4.0,but I hava a llama server on the Cloudflare.
Maybe we could use the llama to generate the code.

Closing in favor of https://github.com/abi/screenshot-to-code/issues/17


Shell file not found: self_supervised/BYOL-ACDC.sh

Hello,
I did not find this file in repositories: self_supervised/BYOL-ACDC.sh
What is it used for? Can you provide the file?

The self_supervised/BYOL-ACDC.sh is no longer needed.
To run the BYOL pretraining you need to modify the self_supervised/configs/byol_acdc.yaml#L32 yaml file then run the pretraining as explained in the README's Pretraining section.


chore: update crane

shouldn't flake.lock also change?

Right... . Why it didn't automatically... ?


Start-ADOPSPipeline -Branch for non refs/heads/

**Is your feature request related to a problem? Please describe. **
Start-ADOPSPipeline -Branch 'refs/tags/1.1.1' should be allowed but at the moment the input is always prefixed with refs/heads/. Another use case is re-run a pipeline run from a pull request (very rare but hard to fix)
Describe the solution you'd like
If the branch parameter starts with 'refs' use the parameter value else keep the original behavior.

I absolutely see where this could be needed, and can fix it easily. The important part is letting the current behaviour be unchanged to prevent a major bump.
I'll fix it ASAP (before the weekend at least..)


No X3G file exporting?

Hi, so I just got a FF Creator Pro with dual extruders, and I'm trying to use PrusaSlicer with it for its painting functionality.
I am fairly certain that I followed the instructions exactly, but upon exporting a GCode, there is no X3G file being created alongside it. The debug file and the momentary popup after exporting both say everything is functioning, but I am not getting an X3G file out of it.

Windows 10
PrusaSlicer 2.5.0
GPX 2.6.8
Strawberry <IP_ADDRESS>
Everything is freshly installed, literally. I just built this PC 2 days ago for use with 3D stuff.
bunnytest.zip

Try updating to the current make_fcp_x3g.pl script and run it with -vc argument in a command prompt as explained in the troubleshooting section of the README. This will now output more information.

Config read from 'C:\Users\atemu\Documents\3D Printing Files\FFCP-GCodeSnippets-master\make_fcp_x3g.txt'
GPX will not be used because GPX path is not configured
DUALSTRUDE_SCRIPT will not be used because not configured
PWM_SCRIPT will not be used because not configured
RETRACT_SCRIPT will not be used because not configured
All checks seem OK!

I may be too dumb for this, I'm sorry. I don't know how to configure the GPX path, but I understand now that that is what's needed to convert the file.

You have gotten pretty far for someone who claims not to understand things. The only thing missing is the correct line in the make_fcp_x3g.txt file specifying the path to the GPX executable.
You can obtain this path by opening the folder where you installed GPX, holding down the Shift key, and then right-clicking on gpx.exe. This will reveal a normally hidden “Copy as path” menu option, select this and the path to GPX will be in your clipboard.
Then open the make_fcp_x3g.txt file in a text editor, and paste the path after the line starting with “GPX =”. Remove the # character at the start of this line if there is one. In other words, the line should look something like this:
GPX = "C:\Program Files\gpx-2.6.8-win64\gpx.exe"

but the actual path in your case may be different, depending on where you installed GPX. Then save the txt file and try again.

AAAA it's working! Thank you so much for helping me. I'll be sure to donate to you on your website, too.
You have gotten pretty far for someone who claims not to understand things.
My only saving grace is being able to mostly follow instructions. I've done similar UI modding to MMOs before using other people's mods and instructions, but sometimes I hit a snag and need help. OTL


Temporarily Fix Google Basemap

Overview
The API version to which we have manually set our Google basemap
has been deprecated for some time. When the requested version is
unavailable, the API defaults to returning the most current version
(currently, '3.48.1a', which can be seen by entering
'google.maps.version' in the app console).
The Google basemap is currently not loading in staging or production.
Although release notes for 4.8 have not been released, it was scheduled
to release around this time, so it seems likely that the release of
4.8 has some breaking change effecting our basemap.
Manually setting the version to 4.7 (a previous but still-available
version) results in the basemap loading as expected.
Version 4.7 should be available for at least the next quarter (per
https://developers.google.com/maps/documentation/javascript/versions,
it should be available for 3-6 months), which will give us time to
update our code to fix the basemap on current versions.
Connects #1657
Demo
Develop (temporarily fixed):

Production (broken):

Testing Instructions

On develop, run ./scripts/server and note that the basemap is not displayed. There are multiple errors in the console. Enter google.maps.version and note the version number is 4.8.
On this branch, run ./scripts/server and note that the errors are gone and the basemap loads. Enter google.maps.version and note the version number is 4.7.

Checklist

[ ] fixup! commits have been squashed
[ ] CI passes after rebase
[ ] CHANGELOG.md updated with summary of features or fixes, following Keep a Changelog guidelines

Taking a look.

Thanks for the review!


Retry WebElement.click if Exception is thrown

Sometimes, an element may be covered by another, but only for a limited amount of time. This change adds a retry mechanism to a click attempt.

@jsfehler thanks for contribute


added AR opentrons page

added random number generator for AR opentrons page (hidden)
updated instructions for the PCR, unit conversion, and cutting (restriction enzyme) reaction calculators

hidden ar opentron html page


What is the best practice to read date type value?

With cellDates read option, xlsx library tries to convert date-type cell to js Date object.
However, it does not seem to respect date1904 property of sheet when constructing js Date object. https://github.com/SheetJS/js-xlsx/issues/126
excel_date.xlsx
const xlsx = require('xlsx');
const ws = xlsx.readFile('./excel_date.xlsx', {cellDates: true});
console.log('date1904:', ws.Workbook.WBProps.date1904);
const firstSheet = ws.Sheets[ws.SheetNames[0]];
console.log(xlsx.utils.sheet_to_json(firstSheet));

The above code with the attached excel file gives the following result:
date1904: true
[ { Date: 2014-12-30T14:59:08.000Z,
    String: 'I am text',
    number: 1 },
  { Date: '2019-01-01', String: 1, number: 2 },
  { Date: 2014-12-30T14:59:08.000Z, String: '3', number: 3 },
  { Date: 2014-12-30T14:59:08.000Z, String: 2, number: 4 } ]

I expected that the generated js Date objects are of '2019-01-01', but they are skewed due to date1904 problem.
I converted all js Date values in my program.
But I think It would be better that the library do this magical conversion so that users do not need to consider date1904 anymore.
Am I missing useful option?

I am also experiencing this, the dates are wrong already in the workbook returned by readFile. This must be a bug since it breaks the cellDates functionality, the dates in the workbook can't be relied upon due to this issue.
I am having trouble finding a good workaround for this. Could you explain your workaround @jngbng? Did you manually add 4 years to dates in the workbook before converting it to JSON when date1904 is true?

@zoeesilcock There are two more problems. SSF module output, instead of JS native Date, is preferable to represent date when importing excel file.
TL;DR. I am using following workaround code.
// Take following code from<EMAIL_ADDRESS>// They are private scoped and inaccessible from outside of the library.
const basedate = new Date(1899, 11, 30, 0, 0, 0);
const dnthresh = basedate.getTime() + (new Date().getTimezoneOffset() - basedate.getTimezoneOffset()) * 60000;

const day_ms = 24 * 60 * 60 * 1000;
const days_1462_ms = 1462 * day_ms;

function datenum(v, date1904) {
  let epoch = v.getTime();
  if (date1904) {
    epoch -= days_1462_ms;
  }
  return (epoch - dnthresh) / day_ms;
}

function fixImportedDate(date, is_date1904) {
  // Convert JS Date back to Excel date code and parse them using SSF module.
  const parsed = xlsx.SSF.parse_date_code(datenum(date, false), {date1904: is_date1904});
  return `${parsed.y}-${parsed.m}-${parsed.d}`;
  // or
  // return parsed;
  // or if you want to stick to JS Date,
  // return new Date(parsed.y, parsed.m, parsed.d, parsed.H, parsed.M, parsed.S);
}

function useSSFOutput() {
  const wb = xlsx.readFile('./tz_test_dates.xlsx', {cellDates: true});
  const sheet = wb.Sheets[(wb.SheetNames[0])];
  // original output
  const converted = xlsx.utils.sheet_to_json(sheet, {header: 1, cellDates:true});
  // apply hotfix
  const is_date1904 = wb.Workbook.WBProps.date1904;
  const fixed = converted.map((arr) => arr.map((v) => {
    if (v instanceof Date) {
      return fixImportedDate(v, is_date1904);
    } else {
      return v;
    }
  }));
  console.log(fixed.map(arr => arr.map(v => v.toString())));
}

useSSFOutput();

Run above code with tz_test_dates.xlsx and will get following result:
tz_test_dates.xlsx preview:
2019-01-01 | 1960-01-01 | 1908-01-01 |   |   |
-- | -- | -- | -- | -- | --
2019-01-01 | 2019-03-01 | 2019-05-01 | 2019-07-01 | 2019-09-01 | 2019-11-01
[ [ '2019-1-1', '1960-1-1', '1908-1-1' ],
  [ '2019-1-1',
    '2019-3-1',
    '2019-5-1',
    '2019-7-1',
    '2019-9-1',
    '2019-11-1' ] ]

Detail
sheet_to_json uses following code to convert Excel date code to JS Date object.
var basedate = new Date(1899, 11, 30, 0, 0, 0); //<PHONE_NUMBER>000
var dnthresh = basedate.getTime() + (new Date().getTimezoneOffset() - basedate.getTimezoneOffset()) * 60000;

function numdate(v) {
	var out = new Date();
	out.setTime(v * 24 * 60 * 60 * 1000 + dnthresh);
	return out;
}

issue 1: precision bug. refer to #1470
On some countries, you may lose some time (in Korea, -52 sec when parsing).
The problem is that Date.getTimezoneOffset() is not precise enough. (SheetJS/ssf#38)
function showOriginal() {
  const wb = xlsx.readFile('./tz_test_dates.xlsx', {cellDates: true});
  const sheet = wb.Sheets[(wb.SheetNames[0])];
  // original output
  const converted = xlsx.utils.sheet_to_json(sheet, {header: 1, cellDates:true});
  console.log('showOriginal:');
  console.log(converted.map(arr => arr.map(v => v.toString())));
}

showOriginal();

/////////////////////////////////////

function getTimezoneOffsetMS(date) {
  var time = date.getTime();
  var utcTime = Date.UTC(date.getFullYear(),
                         date.getMonth(),
                         date.getDate(),
                         date.getHours(),
                         date.getMinutes(),
                         date.getSeconds(),
                         date.getMilliseconds());
  return time - utcTime;
}

const importBugHotfixDiff = (function () {
  const basedate = new Date(1899, 11, 30, 0, 0, 0);
  const dnthreshAsIs = (new Date().getTimezoneOffset() - basedate.getTimezoneOffset()) * 60000;
  const dnthreshToBe = getTimezoneOffsetMS(new Date()) - getTimezoneOffsetMS(basedate);
  return dnthreshAsIs - dnthreshToBe;
}());

function fixPrecisionLoss(date) {
  return (new Date(date.getTime() - importBugHotfixDiff));
}

function showPrevisionLossHotfix() {
  const wb = xlsx.readFile('./tz_test_dates.xlsx', {cellDates: true});
  const sheet = wb.Sheets[(wb.SheetNames[0])];
  // original output
  const converted = xlsx.utils.sheet_to_json(sheet, {header: 1, cellDates:true});
  // apply hotfix. ignore date1904 problem for now.
  const fixed = converted.map((arr) => arr.map((v) => {
    if (v instanceof Date) {
      return fixPrecisionLoss(v);
    } else {
      return v;
    }
  }));
  console.log('showPrevisionLossHotfix:');
  console.log(fixed.map(arr => arr.map(v => v.toString())));
}

showPrevisionLossHotfix();

Run above code after setting computer's time zone to Asia/Seoul (UTC+09:00) then will get:
showOriginal:
[ [ 'Mon Dec 31 2018 23:59:08 GMT+0900 (Korean Standard Time)',
    'Thu Dec 31 1959 23:29:08 GMT+0830 (Korean Standard Time)',
    'Tue Dec 31 1907 23:27:00 GMT+0827 (Korean Standard Time)' ],
  [ 'Mon Dec 31 2018 23:59:08 GMT+0900 (Korean Standard Time)',
    'Thu Feb 28 2019 23:59:08 GMT+0900 (Korean Standard Time)',
    'Tue Apr 30 2019 23:59:08 GMT+0900 (Korean Standard Time)',
    'Sun Jun 30 2019 23:59:08 GMT+0900 (Korean Standard Time)',
    'Sat Aug 31 2019 23:59:08 GMT+0900 (Korean Standard Time)',
    'Thu Oct 31 2019 23:59:08 GMT+0900 (Korean Standard Time)' ] ]
showPrevisionLossHotfix:
[ [ 'Tue Jan 01 2019 00:00:00 GMT+0900 (Korean Standard Time)',
    'Thu Dec 31 1959 23:30:00 GMT+0830 (Korean Standard Time)',
    'Tue Dec 31 1907 23:27:52 GMT+0827 (Korean Standard Time)' ],
  [ 'Tue Jan 01 2019 00:00:00 GMT+0900 (Korean Standard Time)',
    'Fri Mar 01 2019 00:00:00 GMT+0900 (Korean Standard Time)',
    'Wed May 01 2019 00:00:00 GMT+0900 (Korean Standard Time)',
    'Mon Jul 01 2019 00:00:00 GMT+0900 (Korean Standard Time)',
    'Sun Sep 01 2019 00:00:00 GMT+0900 (Korean Standard Time)',
    'Fri Nov 01 2019 00:00:00 GMT+0900 (Korean Standard Time)' ] ]

Notice that 52 seconds error has gone, but '1960-01-01' and '1908-01-01' are not correctly parsed.
It is due to following issue.
issue2: timezone offset is not constant within one time zone.
Noice that dnthresh depends on the timezone offset of CURRENT TIME.
But on some countries, timezone offset changes (or have changed) over time.
In Korea, it is GMT+09:00 now, but it was GMT+08:30 in 1960 and GMT+08:27 in 1908.
In Los Angeles in US, it is GMT-08:00 in January and GMT-07:00 in October due to summer time.
For these countries, dnthresh should not be constant and we should consider time zone change.
SSF module which is timezone-agnostic rescues us.
// --------------------------------------------------
// Take following code from<EMAIL_ADDRESS>// They are private scoped and inaccessible from outside of the library.
//
const basedate = new Date(1899, 11, 30, 0, 0, 0);
const dnthresh = basedate.getTime() + (new Date().getTimezoneOffset() - basedate.getTimezoneOffset()) * 60000;

const day_ms = 24 * 60 * 60 * 1000;
const days_1462_ms = 1462 * day_ms;

function datenum(v, date1904) {
  let epoch = v.getTime();
  if (date1904) {
    epoch -= days_1462_ms;
  }
  return (epoch - dnthresh) / day_ms;
}
// -------------------------------------------------

function fixImportedDate(date, isDate1904) {
  const parsed = xlsx.SSF.parse_date_code(datenum(date, false), {date1904: isDate1904});
  // return `${parsed.y}-${parsed.m}-${parsed.d}`;
  return new Date(parsed.y, parsed.m, parsed.d, parsed.H, parsed.M, parsed.S);
}

function useSSFOutput() {
  const wb = xlsx.readFile('./tz_test_dates.xlsx', {cellDates: true});
  const sheet = wb.Sheets[(wb.SheetNames[0])];
  // original output
  const converted = xlsx.utils.sheet_to_json(sheet, {header: 1, cellDates:true});
  // apply hotfix
  const isDate1904 = wb.Workbook.WBProps.date1904;
  const fixed = converted.map((arr) => arr.map((v) => {
    if (v instanceof Date) {
      return fixImportedDate(v, isDate1904);
    } else {
      return v;
    }
  }));
  console.log('useSSFOutput:');
  console.log(fixed.map(arr => arr.map(v => v.toString())));
}

useSSFOutput();

Above code gives following result:
On LosAngeles timezone:
showPrevisionLossHotfix:
[ [ 'Mon Dec 31 2018 23:00:00 GMT-0800 (Pacific Standard Time)',
    'Thu Dec 31 1959 23:00:00 GMT-0800 (Pacific Standard Time)',
    'Tue Dec 31 1907 23:00:00 GMT-0800 (Pacific Standard Time)' ],
  [ 'Mon Dec 31 2018 23:00:00 GMT-0800 (Pacific Standard Time)',
    'Thu Feb 28 2019 23:00:00 GMT-0800 (Pacific Standard Time)',
    'Wed May 01 2019 00:00:00 GMT-0700 (Pacific Daylight Time)',
    'Mon Jul 01 2019 00:00:00 GMT-0700 (Pacific Daylight Time)',
    'Sun Sep 01 2019 00:00:00 GMT-0700 (Pacific Daylight Time)',
    'Fri Nov 01 2019 00:00:00 GMT-0700 (Pacific Daylight Time)' ] ]
useSSFOutput:
[ [ 'Fri Feb 01 2019 00:00:00 GMT-0800 (Pacific Standard Time)',
    'Mon Feb 01 1960 00:00:00 GMT-0800 (Pacific Standard Time)',
    'Sat Feb 01 1908 00:00:00 GMT-0800 (Pacific Standard Time)' ],
  [ 'Fri Feb 01 2019 00:00:00 GMT-0800 (Pacific Standard Time)',
    'Mon Apr 01 2019 00:00:00 GMT-0700 (Pacific Daylight Time)',
    'Sat Jun 01 2019 00:00:00 GMT-0700 (Pacific Daylight Time)',
    'Thu Aug 01 2019 00:00:00 GMT-0700 (Pacific Daylight Time)',
    'Tue Oct 01 2019 00:00:00 GMT-0700 (Pacific Daylight Time)',
    'Sun Dec 01 2019 00:00:00 GMT-0800 (Pacific Standard Time)' ] ]

On Asia/Seoul timezone:
showPrevisionLossHotfix:
[ [ 'Tue Jan 01 2019 00:00:00 GMT+0900 (Korean Standard Time)',
    'Thu Dec 31 1959 23:30:00 GMT+0830 (Korean Standard Time)',
    'Tue Dec 31 1907 23:27:52 GMT+0827 (Korean Standard Time)' ],
  [ 'Tue Jan 01 2019 00:00:00 GMT+0900 (Korean Standard Time)',
    'Fri Mar 01 2019 00:00:00 GMT+0900 (Korean Standard Time)',
    'Wed May 01 2019 00:00:00 GMT+0900 (Korean Standard Time)',
    'Mon Jul 01 2019 00:00:00 GMT+0900 (Korean Standard Time)',
    'Sun Sep 01 2019 00:00:00 GMT+0900 (Korean Standard Time)',
    'Fri Nov 01 2019 00:00:00 GMT+0900 (Korean Standard Time)' ] ]
useSSFOutput:
[ [ 'Fri Feb 01 2019 00:00:00 GMT+0900 (Korean Standard Time)',
    'Mon Feb 01 1960 00:00:00 GMT+0830 (Korean Standard Time)',
    'Sat Feb 01 1908 00:00:00 GMT+0827 (Korean Standard Time)' ],
  [ 'Fri Feb 01 2019 00:00:00 GMT+0900 (Korean Standard Time)',
    'Mon Apr 01 2019 00:00:00 GMT+0900 (Korean Standard Time)',
    'Sat Jun 01 2019 00:00:00 GMT+0900 (Korean Standard Time)',
    'Thu Aug 01 2019 00:00:00 GMT+0900 (Korean Standard Time)',
    'Tue Oct 01 2019 00:00:00 GMT+0900 (Korean Standard Time)',
    'Sun Dec 01 2019 00:00:00 GMT+0900 (Korean Standard Time)' ] ]

@jngbng thanks for going over this!  Was not aware of the minute differences in KST.
The date handling is a bit of a mess and we likely have to revisit both SSF and this library.  In the next push, we're going to deprecate the SSF repo and merge the contents as a package in this repo to make it easier to update both at the same time (they are separate because originally the XLS parsing was a separate library and both used this component)
Since you looked into this a bit, maybe you can share your opinion about the best representation for the library:
Excel dates
Under the hood, Excel dates are numbers representing the number of days (+ fraction of a day) from a given epoch.
The default epoch is December 31 1899 (rendered as "1/0/00").  When the 1904 Date System is used, the default epoch is January 1 1904.
As is the case with JS, Excel days are exactly 86400 seconds long, even during DST adjustments.
How Excel deviates from JS

"1900 leap year bug": Excel considers 1900 a leap year when it actually wasn't.  Rule of thumb for leap years: !(year % 4) && ((year % 100) || !(year % 400))  Excel has a fake day with date code 60 ("2/29/00").  Other things like "day of week" were changed for dates before 2/29/00 in order to be consistent.

This is why SSF has a special date struct -- to support the date that JS cannot.

"relative time": The Excel epochs are dates are relative to the computer timezone.  In the 1900 date system, if you store a cell with value 0 and format "yyyy-mm-dd hh:mm:ss", the formatted text will be "1900-01-00 00:00:00" no matter what timezone your computer uses!  JS has a concept of Universal Time.

"timezone": The previous point wouldn't normally be a huge issue, but Excel doesn't store any information about the timezone of the computer that saved the file.  There's no way to know the universal date corresponding to the value.  XLS has a "Country" record, but that's useless for countries like USA which have multiple timezones.

File Dates
Excel has three ways of storing dates in files:

Every Excel format can store a number with a format code that represents a field from a date interpretation (e.g. "yyyy" for 4-digit date)

XLSX has a special date type "d" which can understand ISO8601 strings (2019-10-31T00:00:00.000Z is interpreted as midnight of 2019-10-31)

Plaintext formats like CSV have values that are interpreted by the core Excel engine.

What SheetJS currently attempts to do
The internal representation of date cells is the date as understood in the JS engine timezone.
For example, in ET (currently UTC-400), a date cell like
      <c r="A1" t="d">
        <v>2019-10-31T00:00:00.000Z</v>
      </c>

or a number cell using a date format like
      <c r="A2" s="1">
        <v>43769</v>
      </c>

will be parsed and stored as 43769 if cellDates is not specified OR 2019-10-31T04:00:00.000Z if cellDates is set to true.
The main reason is to make the easier to supply dates from JS.  For example:
var wb = XLSX.utils.book_new();
var ws = { A1: {t:"d", v:new Date("October 31 2019 3:00 PM") }, "!ref": "A1:A1" };
XLSX.utils.book_append_sheet(wb, ws, "Sheet1");
XLSX.writeFile(wb, "test1565.xlsx");

will create a file that stores the date code 43769.625 even though new Date("October 31 2019 3:00 PM") changes depending on the timezone: in ET, new Date("October 31 2019 3:00 PM").valueOf() is<PHONE_NUMBER>000, while it is<PHONE_NUMBER>000 in UTC.
Design goals and considerations
"Date representation": Converting between XLSX type "d" and Excel Date codes should work irrespective of the computer timezone setting.
"Round-trip": Reading a file and writing it back should always store the same date.
"Cross-Timezone": Server-side processing is tricky because the timezone of the user and server oftentimes differ.
"Limited complexity": Limited date shifting (adding or subtracting the timezone offset) in the end-user code
Since you mentioned that the timezone offset changed by a little bit, we'll rethink that part.

I have not looked into this as deeply yet, but it is also a problem for me - ideally I want to interpret dates as UTC, perhaps this could be an option? I realise that you may then run into round-tripping problems. It also seems in new versions as opposed to old ones - say 0.10.0 you can't manually tell whether an XLS cell is a date after reading a workbook?
Naively, as a first solution to the fractional timezone issues, (new Date().getTimezoneOffset() - basedate.getTimezoneOffset()) * 60000 could simply be replaced with new Date().getTime() - basedate.getTime() ?

This thread has ballooned to include a general discussion about handling dates. Could we return to the original issue, namely that readFile doesn't return correct data? As was described in the original issue, the library correctly identifies that the file is based on 1904 rather than 1900 but the dates in the result don't reflect that. Surely this isn't meant to work this way?

I've just been bitten by this—shouldn't we autodetect which date format we have, instead of returning an off-by-4-years date?

@Fil "off by 4 years" sounds like a problem with the date system (1900 vs 1904) which is not related to the issue at hand.  Please raise a new issue and include an example

I don't understand your comment. The issue as described in the OP gives the example of a file with four rows that contain a date of 2019-01-01. When it's opened with xlsx, three of the dates are transformed (on my computer in European timezone) into 2014-12-30T23:59:39, which is off by 4 years, 1 day and 21 seconds. For the OP (evaluated in Korean TZ), the offset is 4 years, 9 hours and 42 seconds.

How about changing the basedate to UTC?
var basedate = new Date(1899, 11, 31, 0, 0, 0);
// var dnthresh = basedate.getTime();
var dnthresh = Date.UTC(1899, 11, 31, 0, 0, 0);
var base1904 = new Date(1900, 2, 1, 0, 0, 0);
function datenum_local(v, date1904) {
var epoch = v.getTime();
if(date1904) epoch -=<PHONE_NUMBER>1000;
else if(v >= base1904) epoch += 246060*1000;
// return (epoch - (dnthresh + (v.getTimezoneOffset() - basedate.getTimezoneOffset()) * 60000)) / (24 * 60 * 60 * 1000);
return (epoch - (dnthresh + v.getTimezoneOffset() * 60000)) / (24 * 60 * 60 * 1000);
}
var basedate = new Date(1899, 11, 30, 0, 0, 0); //<PHONE_NUMBER>000
function datenum(v, date1904) {
var epoch = v.getTime();
if(date1904) epoch -=<PHONE_NUMBER>1000;
// var dnthresh = basedate.getTime() + (v.getTimezoneOffset() - basedate.getTimezoneOffset()) * 60000;
var dnthresh = Date.UTC(1899, 11, 30, 0, 0, 0) + v.getTimezoneOffset() * 60000;
return (epoch - dnthresh) / (24 * 60 * 60 * 1000);
}
var refdate = new Date();
// var dnthresh = basedate.getTime() + (refdate.getTimezoneOffset() - basedate.getTimezoneOffset()) * 60000;
var dnthresh = Date.UTC(1899, 11, 30, 0, 0, 0) + refdate.getTimezoneOffset() * 60000;
var refoffset = refdate.getTimezoneOffset();
function numdate(v) {
var out = new Date();
out.setTime(v * 24 * 60 * 60 * 1000 + dnthresh);
if (out.getTimezoneOffset() !== refoffset) {
out.setTime(out.getTime() + (out.getTimezoneOffset() - refoffset) * 60000);
}
return out;
}

So I have a file that's created in the Norway timezone and I'm in India and when I convert the dates using cellDates: true there's this one day difference in some dates. I'm using node.js.
Dates in the sheet:
24.09.2020
10.09.2020
10.09.2020
10.09.2020
15.09.2020
25.09.2020
Dates after doing this:
const file = xlsx.readFile(filePath, {
    cellDates: true
});

2020-09-24T17:29:49.999Z
2020-09-09T19:29:50.000Z
2020-09-09T19:29:50.000Z
2020-09-09T19:29:50.000Z
2020-09-14T19:29:50.000Z
2020-09-24T19:29:50.000Z
There's a one day difference. The date 10/09/2020 is coming as 09/09/2020. 15th as 14 and 25th as 24.
I'm not sure if the issue is because of the timezone difference or because of the above mentioned problems that I don't understand much.. What should I do to get the date as in the sheet? Is the date I got after reading file in IST ? Should I convert it to Norway time? Also how would you know which time zone the excel date is in and the converted date is in ?

I did some investigation here are my findings

Browsers don't parse dates from spread sheets well
A lot of issues regarding dates: https://github.com/SheetJS/sheetjs/issues?q=is%3Aissue+is%3Aopen+date
Description of excel dates and js and the variance: https://github.com/SheetJS/sheetjs/issues/1565#issuecomment-548526523

Weird date formatting in excel
Left is the broken one from the client and right is the working excel file

Notice the date is one day behind for the broken excel file however it shows as a different date in the excel cell

Solutions

Ignore this issue if it is doesn't occur frequently
Account for hours that are 24:00
Process spreadsheets with python likely more accurate excel parsing

I am having the same issue my excel has 21-Dec-2021 but the returning json shows 44551.
Is there any option that we can pass to sheet_to_json function so that it returns as 21-Dec-2021 without any formating?

Seems like Date parsing isnt good right now, only with different workarounds for post-update the data

i also meet same issue, here is my solution: when xlsx.read is called, when there is a date need to handle, i  just add a '#' to the start and end of the date, to make it a regular string, when i call sheet_to_json, then i just remove the '#' to restore the original date. ex: date = '2022-01-01 12:40:33'， when i call xlsx.read i trun it into '#2022-01-01 12:40:33#', when i call sheet_to_json i remove '#' and change it back to '2022-01-01 12:40:33'.  just turn date to a regular string as 'abcd...' and change it back then.

can you @Lonli-Lokli @yulei1990 please share your workarounds?

how to add '#'

i also meet the same issue, here is my solution: when xlsx.read is called, when there is a date need to handle, i just add a '#' to the start and end of the date, to make it a regular string, when i call sheet_to_json, then i just remove the '#' to restore the original date. ex: date = '2022-01-01 12:40:33'， when i call xlsx.read i trun it into '#2022-01-01 12:40:33#', when i call sheet_to_json i remove '#' and change it back to '2022-01-01 12:40:33'. just turn date to a regular string as 'abcd...' and change it back then. the advantage is that it can store any date format you want, the shortage is that it looks ugly.

how to add ’#‘

i also meet the same issue, here is my solution: when xlsx.read is called, when there is a date need to handle, i just add a '#' to the start and end of the date, to make it a regular string, when i call sheet_to_json, then i just remove the '#' to restore the original date. ex: date = '2022-01-01 12:40:33'， when i call xlsx.read i trun it into '#2022-01-01 12:40:33#', when i call sheet_to_json i remove '#' and change it back to '2022-01-01 12:40:33'. just turn date to a regular string as 'abcd...' and change it back then. the advantage is that it can store any date format you want, the shortage is that it looks ugly.

how to add ’#‘

export function SerializeDateTime(dt){
return '#' + dt + '#'
}
export function DeserializeDateTime(dt) {
if (dt.length > 2) {
var len = dt.length
return dt.slice(1, len - 1)
}
return dt
}

i also meet the same issue, here is my solution: when xlsx.read is called, when there is a date need to handle, i just add a '#' to the start and end of the date, to make it a regular string, when i call sheet_to_json, then i just remove the '#' to restore the original date. ex: date = '2022-01-01 12:40:33'， when i call xlsx.read i trun it into '#2022-01-01 12:40:33#', when i call sheet_to_json i remove '#' and change it back to '2022-01-01 12:40:33'. just turn date to a regular string as 'abcd...' and change it back then. the advantage is that it can store any date format you want, the shortage is that it looks ugly.

how to add ’#‘

export function SerializeDateTime(dt){ return '#' + dt + '#' }
export function DeserializeDateTime(dt) { if (dt.length > 2) { var len = dt.length return dt.slice(1, len - 1) } return dt }
but when i use xlsx.read(data) from file，date is already changed.

i also meet the same issue, here is my solution: when xlsx.read is called, when there is a date need to handle, i just add a '#' to the start and end of the date, to make it a regular string, when i call sheet_to_json, then i just remove the '#' to restore the original date. ex: date = '2022-01-01 12:40:33'， when i call xlsx.read i trun it into '#2022-01-01 12:40:33#', when i call sheet_to_json i remove '#' and change it back to '2022-01-01 12:40:33'. just turn date to a regular string as 'abcd...' and change it back then. the advantage is that it can store any date format you want, the shortage is that it looks ugly.

how to add ’#‘

export function SerializeDateTime(dt){ return '#' + dt + '#' }
export function DeserializeDateTime(dt) { if (dt.length > 2) { var len = dt.length return dt.slice(1, len - 1) } return dt }

but when i use xlsx.read(data) from file，date is already changed.

Following

It seems that there is still no solution for this problem after 4 years. I'm from Vietnam and mine is 30 second deduct

Can somebody help in this,
https://github.com/SheetJS/sheetjs/issues/718#issuecomment-1837622200

Excel 用【文本】格式保存日期 解析就是正常的 如果必须解析【日期】or【时间】组件调用sheet_to_csv （没有秒数差异） 和sheet_to_json 数据拼接起来 日期时间就是完整的了 我感觉组件本身获取的时间戳应该是对的 sheet_to_json时候日期对象里面导致的时间差

It is normal for Excel to save date parsing in [text] format. If it is necessary to parse the [date] or [time] component, calling sheet_to_csv (without any difference in seconds) and combining the data of sheet_to_json, the date and time are complete. I feel that the timestamp obtained by the component itself should be correct. The time difference caused by the date object in sheet_to_json is correct.


Fix zombie eye appearance

About the PR

Changed the way the zombie transformation works so that it properly changes and restores the victim's eye color.
Zombies now get spooky red eyes like they're supposed to, and regain their original eye color when cloned.
Why / Balance

Fixes #19336
Technical details

Instead of the zombie transformation changing and recoloring the target's eye layer, which caused it to bug out and disappear entirely upon cloning, it now changes and restores their eye color directly.
Media

[X] I have added screenshots/videos to this PR showcasing its changes ingame, or this PR does not require an ingame showcase

Changelog

:cl:

fix: Zombies' eyes turn red the way they were supposed to.
fix: When a zombie is cloned, the cloned person's eyes are visible again.

@EmoGarbage404 This looks reasonable to me, but you know more about the zombie system so your review is appreciated.


Watch errors

This tests and fixes #1
A couple things:

If in watch mode, doesn't exit the process. In this way, if you have an error you can see it, then fix it and simplifyify will rebundle, no need to restart the process.
I'm logging the full error if the error is created during the bundling process. I don't think hiding that behind a process.env flag makes sense. Most developers will need that information to fix the error.
I added the test fixture for this to the ignore for linting as it intentionally includes a syntax error.

@BigstickCarpet let me know if this makes sense, happy to rework or update anything if need be.

Hi.  Sorry for the delayed response.  I've been away on vacation, and I'm just now getting through my backlog of emails.
Thanks for submitting this.  I'll take a look at it right now.

Alright.  This has been released as v1.4.0.  Thank you so much for the contribution!

@BigstickCarpet :+1:


Dockerize the server

Branching from main because the monty-mvp repo is following main instead of develop.


Create an internal service for access to pachd's peer port

Adds a ClusterIP service to pachyderm deployments, that exposes pachd's peer port within the cluster only. AFAICS, this service is also not exposed across namespaces; I verified this by spinning up a shell in a pod in a separate namespace and ensuring that this new service was not in the env vars.
This is useful because it allows pods within the same cluster/namespace to use env vars to discover pachd's ip/peer port. Right now, our go and python clients instead rely on PACHD_SERVICE_HOST/PACHD_SERVICE_PORT, which will not work when TLS is enabled. Consequently, this will also allow me to add support for jupyterhub deployments on TLS-enabled pachyderm clusters.
If that isn't clear, the related python-pachyderm may elucidate: https://github.com/pachyderm/python-pachyderm/pull/176

Tagging two reviewers since, AFAICS this should be secure (see PR description), but want to be precautious -- @msteffen has the context, @jdoliner for a second opinion.

This looks fine to me, my only comment would have been on security. I'm don't feel confident saying this is secure and defer to @msteffen he's the one who originally implemented the peer port.


Gulp build ignores .scss files

I am using Pattern Lab Node v1.2.3 on Mac, using the gulp configuration.
Expected Behavior
I would expect everything inside source/css (the style.scss file and everything inside the /scss folder) to get compiled into style.css
Actual Behavior
Nothing gets compiled. I understand this seems to be intentional but:

Why do you distribute and include the scss files if they do not get compiled?
How can I change this behaviour so that the gulp build will consider the scss files?

Shouldn't the style of my components (atoms, molecules, etc) be described in the scss files that are being ignored by the build?
Steps to Reproduce

Added some variables and classes for my atoms in source/css/scss/base/_text.scss
Tried to apply this new classess to an atom in my pattern library
gulp patternlab:serve
No changes

Hello @philapple -  thanks for the issue report. I understand where you are coming from, and have even supported scss compilation in past versions of Pattern Lab Node, but ultimately removed direct support as not core functionality.
Why? Well, maintaining this step in the build chain is a decision best left to users. Shipping with scss is convenient for many reasons, but forcing people to use this preprocessor goes against the tool agnostic mantra you can find on http://patternlab.io/.
What about webpack? What about postcss? It just comes down to limited bandwidth and reducing lock-in for consumers.
Editions are meant to be extended and wholesale re-purposed to a user's needs. This issue also exists to make installation a bit more streamlined, versus shipping with a limited set of files inside source/. It probably wouldn't hurt to remove the scss files for clarity's sake.
One final note - at a recent workshop I helped an attendee add scss compilation to his edition with about 4 minutes of over-the-should encouragement. That's the beauty of the gulp/grunt task system, Pattern Lab is just another task to fit into whatever workflow you already have. I have designs to write a blog post on how to do this - but haven't found the time yet.
Thanks for your thoughts, and I hope you understand the reasoning I've laid out above.


HitBTC cancel order does not work

Hi,
Not sure if this ever worked, trying to cancel order with:
hitbtc.cancel_order(111111)
And the ID is certainly valid as I see it on the web.
POST http://api.hitbtc.com/api/1/trading/cancel_order?nonce=1503164665&apikey=&clientOrderId=111111
Response: {'Content-Type': 'application/x-www-form-urlencoded', 'X-Signature': '', 'User-Agent': '', 'Accept-Encoding': 'gzip, deflate'} {"CancelReject":{"clientOrderId":"111111","cancelRequestClientOrderId":"","rejectReasonCode":"orderNotFound"}}
And from the documentation:
https://hitbtc.com/api#cancelorder
It seems that some required parameters are missing, such as symbol. What do you think?

That's strange, because in their docs they say, that the only required parameter is the clientOrderId...

I'll add some debugging into this, just a second.

Ok, I got it. Looks like it adds all params to the URL instead of putting the clientOrderId into the body. Will fix it very soon.

There are two thing that confuses me:

I can see in the documentation which params are mandatory O_o
It returns clientOrderId in the error and that means that exchange gets it

@TearsOfTheEarth are you sure that you're sending the clientOrderId and not the cancelRequestClientOrderId ?

Absolutely, even created a standalone script with a single command and copy pasted "Order ID" from HitBTC's terminal.

@TearsOfTheEarth I'm sorry what is the HitBTC terminal? I mean, can you create a HitBTC order with ccxt and then send its id to the cancel_order with ccxt ?

This is how they call their web :)
Sample usage at HitBTC site: see https://hitbtc.com/terminal, My Orders tab. Click Cancel button in required order line.
Sure, ran:
buyOrder = exchange.create_limit_buy_order('DOGE/BTC', 1000, 0.000000392)
print(buyOrder['id'])
print(exchange.cancel_order(buyOrder['id']))
orderNotFound and have it alive in the terminal.

Can you please try the following:
buyOrder = exchange.create_limit_buy_order ('DOGE/BTC', 1000, 0.000000392)
id = buyOrder['info']['ExecutionReport']['clientOrderId'] # we will try this one instead of orderId
print(id)
print(exchange.cancel_order(id))

Does it work? If so, then it means they only use client ids, not their own ids to cancel orders.

Yes, works this way 👍

Ok, thx, we will have to fix it for HitBTC then, such a strange API requirements they have... Will update you soon on this.

Ok, this should be fixed and working normally in 1.4.32+.
Please update to the most recent version and use it as you did with your original example:
buyOrder = exchange.create_limit_buy_order('DOGE/BTC', 1000, 0.000000392)
print(buyOrder['id'])
print(exchange.cancel_order(buyOrder['id'])) # this should now work in 1.4.32+

Thank you very much for reporting this!

Sorry, one more minor fix for this issue is coming up in 1.4.33. Please update to 1.4.33 in a couple of minutes.

Done! Fixed in 1.4.33! Thx again!


Implement integration tests for spanner Client

Once we have the spanner stub (#61) and client (#29) implemented, we need to add some integration tests for the Client.

I think we can close this. There are integration tests implemented for Read(), ExecuteSql, Commit, Rollback and RunTransaction.  If we think we need specific integration tests for some overload or another then we should create specific bugs for each case.

I broke this down into a number of issues, one for each member function that needs an integration test.


[BUG]: Stripe.js Attempting to JSON.parse '[object Object]'

What happened?
Describe the issue
When using Stripe.js, there seems to be an issue where it attempts to JSON.parse the string "[object Object]", leading to an error.
Steps to reproduce the issue
Unfortunately, we can only reproduce this issue through observations on Sentry and not through a specific set of steps. It appears to be occurring intermittently without a clear trigger.

Environment
Chrome <IP_ADDRESS> on Windows 10
Reproduction
No response

Without being able to reproduce this even semi-reliably, I'm not sure there's anything we can do here. This could potentially happen for a variety of reasons: intermittent network failures, user browser extensions interfering, etc.
Are you seeing many instance of these? I can find a single instance of this error matching your description, so I am inclined to suspect a network failure result in incomplete message data. If you do find this happening more regularly or you're able to reproduce, please reach out again.
Our support channels are a good place to share with more sensitive/private details if needed: https://support.stripe.com/contact

Thank you for your response. According to the records in Sentry, the earliest occurrence was 18 days ago, and the most recent one was 3 days ago, with a total of 4 incidents. If this issue continues to persist, I will reach out to customer support to determine the necessary steps to address it.

@DanSnow any resolution here? I am seeing this same error, 21 occurrences over the last month happening fairly regularly (~1-2 each day).

@FredKSchott  Unfortunately, no.

@DanSnow is Stripe investigating this? We have been seeing the same error

@brendanm-stripe do you know if there was any resolution here in closing this? We are seeing the same issue

We do have someone who is able to reproduce this issue

@ElizabethSchafle if you're able to share reproduction steps we can follow that lead to this error, please open a new issue to provide those details. I suspect there may be multiple disparate causes, but if there are some determinate conditions it would be great to reduce these.

Its this plugin that seems to have been causing the issue https://chromewebstore.google.com/detail/modheader-modify-http-hea/idgpnmonknjnojddfkpgkljpfnnfcklj?hl=en-US&utm_source=ext_sidebar&pli=1

So it probably its caused by various browser plugins

Ill open another issue!

https://github.com/stripe/stripe-js/issues/609


feat(sql-udf): support basic anonymous sql udf return expression

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
As titled, the current implementation is just treating the return expression as a normal select function body definition, since currently return sql udf will not involve complex syntax.
See the test cases for supported syntax & example use cases.

Checklist

[ ] I have written necessary rustdoc comments
[ ] I have added necessary unit tests and integration tests
[ ] I have added test labels as necessary. See details.
[ ] I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features #7934).
[ ] My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
[ ] All checks passed in ./risedev check (or alias, ./risedev c)
[ ] My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

[ ] My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

[ ] My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

Seems based on #14139?


Should we include scss_lint?

I'm going to try using scss_lint on Fliper, and imo it's a good idea. CSS can be a bit of a free-for-all, and enforcing some kind of style guide would be a good idea. If we do this, we'll have to come up with a standard configuration, of course.

+1 for trying it out on Fliper. Report back here after eval and let's consider adding? Is there any community style guide or default settings of the linter that we can use with good results?
On Tue, Aug 4, 2015 at 11:08 AM, Alexis King<EMAIL_ADDRESS>wrote:

I'm going to try using scss_lint on Fliper, and imo it's a good idea. CSS can be a bit of a free-for-all, and enforcing some kind of style guide would be a good idea. If we do this, we'll have to come up with a standard configuration, of course.
Reply to this email directly or view it on GitHub:
https://github.com/philosophie/suspenders/issues/81

So far, it seems to have been successful, but we've only been using it for a week or so. I'll give a more comprehensive evaluation after we've been using it for a while.
The most comprehensive style guide I've found is sass-guidelin.es. It is self-described as follows:

An opinionated styleguide for writing sane, maintainable and scalable Sass.

From what I can tell, its usage isn't particularly widespread, but it is a comprehensive guide, and I agree with the majority of its policies. The other style guide I found was Chris Coyier's, which is much more lenient and mostly aligns with the default settings of scss-lint.
I think we have two options:

Follow the default configuration of scss-lint as closely as possible. The defaults are mostly sane. I've been using it with four small tweaks to the config, and it's been working well.
Enforce the style guide I mentioned above. This will likely be stricter, and it's incompatible with the default config in a few small ways, but it might be worth trying out.

I'd advocate for trying to use the configuration provided by the above style guide on a project and seeing how it goes. It seems like a good idea to me. For now, we'll probably just keep using the defaults on Fliper to see how that works out.


updated Swift-Radio-Pro icons

added swift diamond icon, plus extra fire icon to represent 1000+ stars on Swift-Radio-Pro

:+1:

Thanks again for the great list!


Livekit URL path suffix

I'm using livekit behind a reverse proxy, and have the traffic being routed to livekit using a suffix path.
Let's say wss://localhost/my/custom/path. I can make Javascript clients use this path, and even the livekit Meet doesn't complain.
However, it seems that Rust SDK (and the SDKs that use it, like python) just ignores my/custom/path and replace it completely with just /rtc
The root cause seems to be in the line.
Can this be changed?

Yes, it seems like the behaviour of the client-sdk-js is to append rtc/
We're currently overriding the path on the Rust SDK, I can push a fix


TAGS do not work correctly with npx uuv e2e command

Describe the bug
I would like to execute only one scenario for a feature test.
So, i set a tag above my scenario like that :
  @mobile
  Scenario: Connexion Portail OK
   ....
   .....

  @desktop
  Scenario: Connexion Portail OK
   ....
   .....

And then i execute this command :
npx uuv e2e --env="{'TAGS':'@mobile'}"
But uuv executes the both tests.
Starting the run with 2 tests
  Feature: uuv\e2e\login-test.feature
    ✓ Connexion Portail OK  (5496ms)
    ᙭ Connexion Portail KO  (6916ms)
End of the tests execution

Expected behavior
Only one scenario executed
Desktop (please complete the following information):

OS: Window
Browser chrome

In fact, this is not a bug but rather a feature that has not been implemented for @uuv/playwright. I'll look into how to implement it.


Array Express Downloader

This PR adds the array_express downloader. This downloader is a Celery task which takes a DownloaderJob ID. It then looks up the batch for the job, determines the source URL and destination URL, and downloads the file to it. I've created a Django command to setup the required database records and to call the task to test it. I didn't do this in a unit test because the task doesn't do much other than interact with Array Express and I don't want to download a file from them every time the full test suite is run.
I have tested this by running ./run_worker.sh and ./run_tester.sh and verifying that the file was downloaded to workers/volume/expression_data/array_express/A-AFFY-1/E-GEOD-96.raw.1.zip.
I also tested this by putting raise Exception("test") within the try block of the task and made sure that the logging was properly formatted and that the job was marked as a failure.
Many of the files changed are not related to this functionality because I am improving the configuration of the project incrementally as I work on it. I also upgraded the python version to 3.6.1.

Ok @rzelayafavila I've made some changes. Whaddaya think?


How to manually add annotation by hand after each epoch?

Assuming baal returns a list of datums to be labeled by humans after each epoch, how do I connect it to Amazon Mechanical Turk to get them actually labeled for the next epoch's training? Can the training process be stalled while humans label these examples?

Hello,
I'll rework this tutorial, but we do have a tutorial on how to use BaaL for production.
I'm not knowledgeable in Amazon Turk, but I think you could have a loop similar to the script below. In this case, get_label_from_amazon_turk would block until we have the label.
Let me know how we can help! We would gladly improve our API to better suit your use case.
baal_model : ModelWrapper = ...
active_learning_ds: ActiveLearningDataset = ...
heuristic : BALD = ...
QUERY_SIZE = 10 # Amount of label to get between retraining.
NUM_STEPS = 100 # How many round of active learning.

for step in range(NUM_STEPS):
  # 2. Train the model for a few epoch on the training set.
  print(f"Training on {len(active_learning_ds)} items!")
  baal_model.train_on_dataset(active_learning_ds, optimizer, batch_size=16, epoch=5, use_cuda=USE_CUDA)
  baal_model.test_on_dataset(test_dataset, batch_size=16, use_cuda=USE_CUDA)

  print("Metrics:", {k:v.avg for k,v in baal_model.metrics.items()})

  # 3. Select the K-top uncertain samples according to the heuristic.
  pool = active_learning_ds.pool
  if len(pool) == 0:
    print("No items to label.")
    break

  predictions = baal_model.predict_on_dataset(pool, batch_size=16, iterations=15, use_cuda=USE_CUDA, verbose=False)
  top_uncertainty = heuristic(predictions)[:QUERY_SIZE]
  # Translate the indices back to the full dataset.
  raw_top_uncertainty = active_learning_ds._pool_to_oracle_index(top_uncertainty)
  # 4. Label those samples.
  labels = get_label_from_amazon_turk([train_dataset.files[idx] for idx in raw_top_uncertainty])
  active_learning_ds.label(top_uncertainty, labels)

Hi @Dref360 , I am working on a similar (school) project, where I have a front-end for Image Labelling (kind-of Label Studio) and I am thinking on using BAAL for the Active Learning in the back-end.
The solution I have in mind is to, rather than having a loop (NUM_STEPS), I would convert it to a function (the loop) to be called every time the Annotator (from the front-end) returns the labelled images.
The difference in my function (compared to the current loop) is that I would load the model (from the saved checkpoint) to load the new labelled data and save the new checkpoint at the end of this function (after sending the new unlabelled data to the Annotator's queue).
I do have three questions, I am completely new on Machine Learning and specifically on Active Learning.

since I am saving the current state of the model (with torch.save), does it affect in some way to resume the training on a different moment? (equivalent to run the loop once, and then re-loading the model to re-run the loop once again, compared to run the loop continuously as in the example provided).
I have been reading your documentation (and torch lightning docs, and lightning flash docs) and have a hard time understanding how to actually label manually, like all the tutorial focus on a research use-case, but don't give any "hints" on how to feed manual labels to the codes given as example. Maybe I am missing something. The example "how to use BaaL for production." is the only reason I understood how to use a manual labelling function.

continuing on the second point.
I wanted to use lighting flash in my project (to use with BAAL), but I have spent the whole day digging into the source code to understand have can I implement a Manual Label (Oracle) function. in Lightning Flash, the BAAL framework is so much wrapped up that I don't think is easy to add an Oracle function.
I am talking about this tutorial:
https://lightning-flash.readthedocs.io/en/latest/integrations/baal.html
# 3.2 Create the active learning loop and connect it to the trainer active_learning_loop = ActiveLearningLoop(label_epoch_frequency=1) active_learning_loop.connect(trainer.fit_loop) trainer.fit_loop = active_learning_loop
Can you point me in the right direction on how may I insert an Oracle function (to manually label data) by using Lightning Flash with BAAL?

my third question is, it is intentional that "ActiveLearningLoop" class do not have a flag (or an easy way) to allow manual annotation?
in the "step" method, you could add a flag to allow the user to manually annotate the data.

               if len(to_label) > 0:                     self.dataset.label(to_label[: self.query_size])                     return True
or is this class just meant for demonstration?
Thanks and sorry for the long text.

Hi!
Yeah the Lightning Flash integration is mostly for research right now, but you could implement your own ActiveDataModule and change the label function here. I think the Flash team is working with Label Studio right now to get the "production" side going. We are currently not involved in this part of the project.
To answer your questions,

It shouldn't pose an issue as we keep track of what is labelled in ActiveLearningDataset.state_dict. You might need to save the actual labels yourself thought.
I think "How to use BaaL for Prod" is a good way to start. You can use Pytorch Lightning to train, evaluate and get predictions. You then have your labelling function that ask for labels for the most uncertain examples.
ActiveLearningLoop is made for research currently. We could add methods to support your usecase. I'm thinking something like ActiveLearningLoop.get_indices_to_label that does not call label, but simply returns the indices. Would that work?
We are really open to modifying our API to support your usecase!

Don't hesitate to join our Slack for easier communication.

@Dref360 can we close this issue safely?

Yes


[New game]: bad luck square

🎮 Game Request
The Bad Luck Square is a simple shooter game that will challenge your skills.
Point down the features
WASD to move.
Arrow keys (or mouse) to shoot.
You'll have to protect a square by shooting enemies and buying upgrades.
Select program in which you are contributing
GSSoC23
Code of Conduct

[X] I follow CONTRIBUTING GUIDELINE of this project.

Hey @S-ishita !
Thank you for raising an issue 💗
You can self assign the issue by commenting /assign in comment 😀
Make sure you follow CODE OF CONDUCT and CONTRIBUTING GUIDELINES 🚀
Don’t Forget to ⭐ our GameZone🎮
Make sure you join our Discord🕹

Hey @S-ishita ! Thank you so much for your raising the issue💗
It’s all yours, you can come anytime again and make some contributions! 🚀
Alone, we can do little, but together we can do so much! 😇


refactor(ci): meta backend agnostic e2e tests [WIP]

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?

Checklist

[ ] I have written necessary rustdoc comments
[ ] I have added necessary unit tests and integration tests
[ ] I have added test labels as necessary. See details.
[ ] I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features #7934).
[ ] My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
[ ] All checks passed in ./risedev check (or alias, ./risedev c)
[ ] My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

[ ] My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

[ ] My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#19568  👈 (View in Graphite)
#19560
main

This stack of pull requests is managed by Graphite. Learn more about stacking.


Update contact.rb

The contact list id is optional so it should be optional

Thanks! Will cut a new release shortly.


Behavior clarification is needed for pmpcfg.L==1 and pmpcfg.A==0

In section 3.6.1 (PMP CSRs), Locking and Privilege Mode subsection, it is not clear what the behavior is when pmpcfg.L==1 and pmpcfg.A==0.
Could we add a sentence to clarify the behavior under this condition?

What is unclear? The A and L fields are orthogonal. It’s just a PMP
register that is unmodifiable and matches no addresses.
On Thu, Sep 26, 2019 at 1:49 PM chuanhua<EMAIL_ADDRESS>wrote:

In section 3.6.1 (PMP CSRs), Locking and Privilege Mode subsection, it is
not clear what the behavior is when pmpcfg.L==1 and pmpcfg.A==0.
Could we add a sentence to clarify the behavior under this condition?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/riscv/riscv-isa-manual/issues/449?email_source=notifications&email_token=AAH3XQTO7TF4DG7F4F5A4KDQLSOURA5CNFSM4I2Y6AM2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HN3UUJQ,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAH3XQQ7BBXKO2XYHKVK4ATQLSOURANCNFSM4I2Y6AMQ
.

Why the A and L fields are orthogonal? When a PMP entry is off (A==0), the L field should have no effect, right? Otherwise, once a L bit is set, how can I turned on the entry to setup more protections?

The reset values for L and A fields are not specified in the spec, so it is possible that after reset, L is set to 1 and A is set to 0.

The Reset section specifies that A=0 and L=0 after reset: https://github.com/riscv/riscv-isa-manual/blob/efd8a03217843966d3e9a139c321b97d7725b1b6/src/machine.tex#L2462

Thanks for pointing me to the reset section. I missed it. But, the important point is that at least it is not clear to me that the L and A field are orthogonal. Usually, from my experience, when the enable field of a table entry is disabled, the other fields in the entry will have no effect. So, could you clarify this point in the spec to reduce the possible misunderstanding about the relationship between the A and the L fields?

I appreciate the feedback that this is unclear.  I will fix it.


Version nextclade binary for transition, i.e. nextclade -> nextclade2

Soon we will release nextclade v3. Once nextclade v3 hits bioconda, workflows that rely on v2 syntax will break. To prevent this, this PR makes it explicit that the syntax used is for v2 and that the v2 binary nextclade2 should be used.
This binary is in both the nextstrain-base docker image and in the conda-base managed conda environment.
Users who don't use either will have to add nextclade2 to the path.
Over the next few weeks, we will transition repos to use nextclade3. The reason to add nextclade2 now is so that workflows don't break if they haven't yet transitioned to nextclade3 at the time of v3 release. It also makes it easy to search which workflows need to be changed.

Merging as we will release tomorrow

The ingest pipeline ran fine the last couple days but ran into an error today for the align rule:
[batch] [2024-01-18T08:13:06-08:00] Error:
[batch] [2024-01-18T08:13:06-08:00]    0: ^[[91mspecified file not found in archive^[[0m
[batch] [2024-01-18T08:13:06-08:00] Location:
[batch] [2024-01-18T08:13:06-08:00]    ^[[35mpackages_rs/nextclade-cli/src/dataset/dataset_download.rs^[[0m:^[[35m65^[[0m
[batch] [2024-01-18T08:13:06-08:00] Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
[batch] [2024-01-18T08:13:06-08:00] Run with RUST_BACKTRACE=full to include source snippets.

It's not clear why this ran fine before, but I think this is due to the nextclade dataset command downloading the V3 dataset?
https://github.com/nextstrain/mpox/blob/5d3a2675b5f3c2c142d8d65dd8937f093e9110be/ingest/workflow/snakemake_rules/nextclade.smk#L7


Make linux pypy wheels manylinux compliant

Great stuff! Thanks for the clarifications as well. :smile:


Evaluate whether to create tests to ensure browser compatibility

If it makes sense, then run tests in CI as well.

Seems excessive to have this kind of test, at least for the monent, closing.


Fix youtube video link in RTD

Since new users of ADS might find trouble in installing  ADS  or even FSLeyes plugin, a youtube video link detailing the process of installing was required. We were already had a video but it was only present in the README.md and was missing from RTD. This PR aims to fix the mising link in RTD and provide instructions for the same.
Fix #378

@jcohenadad , I've made the suggested changes and this PR is ready for review.

@vs74 i'm getting this in the rendered doc:

@jcohenadad, there was some syntax error in the iframe tag. I've fixed it.
here is the output of RTD

can we merge?

can we merge?

We can. I was just waiting for the tests on travis-CI to complete.


Options to apply bar border to specific sides of the bar

What problem does this feature solve?
When adding a border to the bars in a bar chart, I expect to see a border on 3 sides (top, left, and right), not all 4. I prefer the bottom border to be left empty for 2 reasons:

Avoid having a border sitting on top of the axis baseline (looks messy)
Avoid border overlap when creating a stacked bar chart (the top border of each bar should act as the bottom border of the bar above it)

This is a similar problem to border radius, where it was solved using an array to specify each side. I think that could work for border width too.
What does the proposed API look like?
itemStyle: {
    borderWidth: [0, 1, 1, 1]
}

+1 Needing this feature to achieve design like this:

+1 here

+1


Plutus PAB chain index issue

Summary
my goal is to deploy plutus smart contract into cardano testnet using PAB. For using PAB I have synced my testnet cardano node and wallet. Now when I was trying to sync chain index it sync's till 92% then it starts resyncing from 71.86%. I need help of you guys. Thankyou in advance.
reference I used to do:
YT video: https://www.youtube.com/watch?v=X6AyZIZ0vaE
github: https://github.com/input-output-hk/plutus-apps/blob/main/plutus-pab/test-node/README.md
Steps to reproduce the behavior

Install cardano-node & cardano-wallet
cabal build plutus-pab-examples plutus-chain-index
./start-testnet-node.sh
cardano-wallet serve
--testnet testnet/testnet-byron-genesis.json
--node-socket testnet/node.sock
cabal exec -- plutus-chain-index --config testnet/chain-index-config.json start-index

Actual Result
chain index just simple pause syncing around 92% in first time of doing and in second time it started resyncing from around 71%
Expected Result
chain index should to properly synced for further application
Describe the approach you would take to fix this
I have deleted the whole data in plutus chain index db and then tried to resync it (stops around 92% then restarts syncing from around 71%)
System info
OS: [ubuntu 20.04]
RAM: [16 GB]
Disk Capacity: [512.1 GB]

warra support

Hi! We had the same issue, we did a workaround by replacing the chain index with blockfrost. It is implemented in this PR:
https://github.com/input-output-hk/plutus-apps/pull/655

Plutus-apps no longer uses the testnet node version because that network is being decommissioned.
This integration test shall be reconfigured to use one of the replacement testnets: preview pre-prod
See here for all networks and their configs: https://book.world.dev.cardano.org/environments.html

thankyou @james-iohk i have changed my configuration to pre-production and now its working!
but during the pab db migrate i faced below issue can you please help
token-pab: SQLite3 returned ErrorNotADatabase while attempting to perform prepare "SELECT name, sql from sqlite_master where type='table'": file is not a database

It was my mistake I have changed my db config data. Just kept db config data as it is and now its working! thankyou all


Children components of render functions

I'm using vue-meta on a children component of a render function that use createElement to render a component List with different props given by routes.
The method beforeMount is called everytime I change the route (tried to log on it), but metaInfo updates only at first render (logged on title() method as suggested by docs), how is it possible? If i use metaInfo on component with render function everything is working.
Thank you in advice.

Hi @liqueflies - thank you for giving vue-meta a try. I'm not exactly sure what you're asking here - are you experiencing a bug? Is the metaInfo of the child component not being included in your data?

Okay maybe I didn't explain what was my problem :)
Two of my routes:
 { path: '/', component: switch('home') },
 { path: '/about', component: switch('about') },

switch function (the render function):
export function switch (page) {
    return {
        name: `section-${page}`,
        render (h) {
            return h(MyChildComponent, {
                props: { page }
            });
        }
    };

MyChildComponent:
export default {
    name: 'my-child-component',
    props: {
        page: VueTypes.string
    },
    metaInfo: {
        title() {
            return this.page;
        }
    }
};

The title of MyChildComponent should show home or about.
But show the correct one only at first app run, not when i change the route clicking a link.
I tried to move metaInfo to switch function and it works:
export function switch (page) {
    return {
        name: `section-${page}`,
        metaInfo: {
           title() {
             return page;
           }
        },
        render (h) {
            return h(MyChildComponent, {
                props: { page }
            });
        }
    };

I think I'm encountering a bug, beacause I logged every method of Vue instance lifecycle and everytime it log the correct message.
Maybe there's a problem in child component when parent has render function.
Maybe there's cache problem with child when parent has render function.
Maybe I am the problem :)
Have you ever encountered this issue?
Thank you so much for vue-meta :)

Out of interest, what version of vue-meta are you using? The one from npm?

Yes, the One from npm.

Try uninstalling it and re-installing using Github as a source:
$ npm install https://github.com/declandewet/vue-meta --save

vue-meta is not yet ready for a release on npm yet - I'm still working on getting it to a full working state.

Hi, yes found now the disclaimer :)
If I install from github, there is no main file, so I have to import from 'vue-meta/src' it gives me Uncaught SyntaxError: Unexpected token import error. Using vue-cli with webpack.
If you know how to handle the error, or I'll wait for stable release on npm.
Keep working on it, really good! :) 👍

If I install from github, there is no main file, so I have to import from 'vue-meta/src' it gives me Uncaught SyntaxError: Unexpected token import error. Using vue-cli with webpack.

That's because the Github version isn't transpiled - The main field points to lib/, so you have two options - compile it with npm run build or add it to your resolver in your Webpack config. Or just wait for the official release :smile:

Following your tips, I simply excluded vue-meta from exclude loaders on webpack, but I'm still encountering the same issue: the child component rendered with a function from parent not switch on router-link but only at first app run. The other lifecycle methods are called everytime i click on link.

I see - since this is pretty edge-case I'm going to add a test case for it after the initial release. Use your workaround for now.

Hey @declandewet
Do you have an ETA for the official release?

@Atinux I'm aiming for either tomorrow or the day after. :smile:

👏
I'm having a prototype of nuxt.js working, missing vue-meta to rock 😄

That's great news @Atinux 🎊
@liqueflies I think I've found the issue - although, it's not a bug in vue-meta. It's to do with the way Vue handles VNodes in render functions - According to the docs, VNodes have to be unique.
In your example, the issue is remedied by making MyChildComponent a factory function that returns a new singleton, like so:
export default function MyChildComponent () {
  return {
    name: 'my-child-component',
    props: {
        page: VueTypes.string
    },
    metaInfo: {
        title() {
            return this.page;
        }
    }
  }
}

And then in your switch component (which I have renamed to view, since switch is a reserved word):
export function switch (page) {
    return {
        name: `section-${page}`,
        render (h) {
            return h(MyChildComponent(), {
                props: { page }
            });
        }
    };
}

I have added an example of this pattern here: https://github.com/declandewet/vue-meta/blob/master/examples/vue-router/app.js
You can test it out by git clone-ing vue-meta and running npm run dev from the root directory - this will start the examples server, then navigate to localhost:8080 and click on "vue-router"

As it turns out, this was being caused by the same logic that causes #10. I've implemented a crude fix in https://github.com/declandewet/vue-meta/commit/6ad56ec73e428316c2e4c115274ca655024c2810
You should be able to use your original syntax now @liqueflies  🎉


MemoryError in ADO build agent when downloading 4+ GB SQL bacpac file from storage account using az storage fs file download

Describe the bug
The az storage fs file download command worked fine a few months ago when I was downloading a 1.9GB sql bacpac file from our storage account to the build machine, but lately the bacpac file has grown to over 4 GB and I'm getting a memory error when trying to download the file to the agent machine.
Related command
az storage fs file download -p database.bacpac  -d C:\database.bacpac -f container --overwrite True --account-name storageaccount --auth-mode login
Errors
ERROR: The command failed with an unexpected error. Here is the traceback:
ERROR:
Traceback (most recent call last):
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 233, in invoke
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 664, in execute
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 729, in _run_jobs_serially
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 721, in _run_job
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/storage/init.py", line 430, in new_handler
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/storage/init.py", line 429, in new_handler
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 698, in _run_job
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 334, in call
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 121, in handler
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/storage/operations/fs_file.py", line 43, in download_file
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/multiapi/storagev2/filedatalake/v2021_08_06/_download.py", line 60, in readall
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/multiapi/storagev2/blob/v2021_08_06/_download.py", line 659, in readall
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/multiapi/storagev2/blob/v2021_08_06/_download.py", line 772, in readinto
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/multiapi/storagev2/blob/v2021_08_06/_download.py", line 131, in process_chunk
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/multiapi/storagev2/blob/v2021_08_06/_download.py", line 154, in _write_to_stream
MemoryError
To check existing issues, please visit: https://github.com/Azure/azure-cli/issues
Issue script & Debug output
az storage fs file download -p database.bacpac  -d C:\database.bacpac -f container --overwrite True --account-name storageaccount --auth-mode login --debug
DEBUG: cli.azure.cli.core.azclierror: Traceback (most recent call last):
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 233, in invoke
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 664, in execute
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 729, in _run_jobs_serially
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 721, in _run_job
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/storage/init.py", line 430, in new_handler
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/storage/init.py", line 429, in new_handler
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 698, in _run_job
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 334, in call
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 121, in handler
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/storage/operations/fs_file.py", line 43, in download_file
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/multiapi/storagev2/filedatalake/v2021_08_06/_download.py", line 60, in readall
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/multiapi/storagev2/blob/v2021_08_06/_download.py", line 659, in readall
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/multiapi/storagev2/blob/v2021_08_06/_download.py", line 772, in readinto
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/multiapi/storagev2/blob/v2021_08_06/_download.py", line 131, in process_chunk
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/multiapi/storagev2/blob/v2021_08_06/_download.py", line 154, in _write_to_stream
MemoryError
ERROR: cli.azure.cli.core.azclierror: The command failed with an unexpected error. Here is the traceback:
ERROR: az_command_data_logger: The command failed with an unexpected error. Here is the traceback:
ERROR: cli.azure.cli.core.azclierror:
Traceback (most recent call last):
Expected behavior
The az storage fs download command should download the bacpac file to the agent build machine.
After that is completely, the import to SQL database command is run using   SqlPackage.exe Action Import...
Environment Summary
azure-cli                         2.57.0
core                              2.57.0
telemetry                          1.1.0
Extensions:
azure-devops                      0.26.0
Dependencies:
msal                              1.26.0
azure-mgmt-resource             23.1.0b2
Python location 'C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe'
Extensions directory 'C:\Program Files\Common Files\AzureCliExtensionDirectory'
Python (Windows) 3.11.7 (tags/v3.11.7:fa7a6f2, Dec  4 2023, 19:24:49) [MSC v.1937 64 bit (AMD64)]
Additional context
No response

Thank you for opening this issue, we will look into it.

Hi @tbrothers, wondering how much free memory is left for the machine/vm at the time of downloading? Would the az storage blob download command work for your use case? It might be a bit faster as well. Still trying to replicate this MemoryError.

I am experiencing the same issue as @tbrothers. I can see the python process memory usage spiking before the MemoryError happens. It appears that az storage fs file download is attempting to download the file into memory entirely before writing it to disk. This is definitely going to cause issues with large files (>1GB in my case).
I just switched over to az storage blob download as suggested by @calvinhzy and this is working perfectly.