NumerBay Model integration

The ExternalCSVs Model object in numerblox handles CSV submission loading from a given directory. This object can be augmented to also support downloading of external submissions from Numerbay.

ExternalCSVs could take a list of Numerbay products as an argument, automatically download them and add to NumerFrame. Another option would be to create a dedicated Model object like NumerBayCSVs.
Include easy Numerbay authentication with Key object.
Add checks to ensure NumerBay downloading is successful and gives clear error message otherwise.

NumerBay: https://numerbay.ai/
NumerBay Python API example: https://docs.numerbay.ai/docs/tutorial-extras/ensemble

Hi @CarloLepelaars ,
I plan to make a PR by this week that will do the following:

add a subclass of ExternalCSVs called NumerBayCSVs in 04_model.ipynb
Init takes data_directory as downloading destination, numerbay_product_names as list of preds to download, numerbay_username and numerbay_password.
during predict, predictions are either loaded from the folder (if already download or placed manually by the user into the folder), or download from NumerBay
add a subclass of  BaseSubmitter called NumerBaySubmitter that takes similar auth args for init, save_csv submits to NumerBay to fulfill orders

Are you/crowdcent ok with the above? Also, I'm not familiar with nbdev, do you require me to write mock test for this (without actual numerbay account)? If writing test is not required, I can make sure the code is correct locally.

Hi @CarloLepelaars ,
Before creating a merge request, I hope you could take a look at the code at here and here for a sanity check. Thanks.
A few things to note:

The submitter took a wrapper aproach that uses a tournament submitter. User can choose to submit to both Numerai and NumerBay or NumerBay only
The NumerBayCSVs currently only supports the classic tournament. The ExternalCSVs also doesn't seem to support Signals, and I wrote some workarounds initially but the code got hacky quickly because the prediction files can be flexible in number of columns, column names and column types. I think these could be better handled in the NumerFrame itself (cleaning, casting and nan-filling for the Signals meta columns). However, I don't have capacity to work on this for this week.
The github workflow is failing by throwing Dataset 'numerai_validation_data.parquet' not available in NumerAPI. I don't have extra time this week to fix this.
Other than above, the two components have been tested locally, the code is included in the notebooks as comments. The underlying numerbay api is tested in the numerbay-api package itself.

Very cool, @restrading!

So do I understand it correctly NumerBaySubmitter will mainly make it easy for prediction sellers to upload to NumerBay and submit their models? Really like this as long as communicate that NumerBay prediction buyers should refer to NumerBayCSVs and go from there.

Good point that ExternalCSVs does not support Signals, because we are only adding the prediction column. I'll have a look if this can be generalized. I'll think about if we can clean this up or abstract away some components in the base class. Great suggestion to handle that within the NumerFrame. I'll definitely check out what we can do there!

Great to hear that it already works well locally, trust you on this!

Do you think it would also be nice to support downloading of Signals data from NumerBay? I saw around 5 listings of these on the website.

Thanks for the reply. Great job for the NumerCon presentation!

Yes. A seller can simply wrap a NumerBaySubmitter outside of the tournament submitters that their already use to submit to both sites. Of course they can also submit to NumerBay only (if they want to separate the pipelines or do re-submissions) by turning off upload_to_numerai.
Yes and I think it might be needed for NumerFrame to introduce some opinions (e.g. always convert friday_date to int if it's a date obj, always fill missing friday_date with the previous friday, etc.)
Thanks but bugs might be inevitable at some point :)
For now I think catering to the prediction blending needs would suffice. There has not been active uploads for the Signals data category. I plan to introduce some Signals data-specific features to NumerBay in the near term to drive usage.

#59 Created a PR


packr: do not reset modTime when creating a file

Problem I ran into
packr here reset the mod time of a file no matter how you have added that file to the box.
What I would expect
packr would store most of the metadata about a file when packing (for in memory) or if reading off disk would take the file info and embed it into its own file object.
Would be glad to throw a PR here if this is acceptable.
Reproduction
This will give you a different mod time every single time you resolve
package main

import (
	"fmt"

	"github.com/gobuffalo/packr/v2"
)

func main() {
	box := packr.New("test", "./test")
	err := box.AddString("binary", "this is a test binary")
	if err != nil {
		panic(err)
	}

	for {
		f, err := box.Resolve("binary")
		if err != nil {
			panic(err)
		}

		info, err := f.FileInfo()
		if err != nil {
			panic(err)
		}

		fmt.Println(info.ModTime())
	}
}

A PR would definitely be welcome.

Mark Bates
On Feb 21, 2019, 11:42 AM -0500, Zach Gershman<EMAIL_ADDRESS>wrote:

Problem I ran into
packr here reset the mod time of a file no matter how you have added that file to the box.
What I would expect
packr would store most of the metadata about a file when packing (for in memory) or if reading off disk would take the file info and embed it into its down file object.
Would be glad to throw a PR here if this is acceptable.
Reproduction
This will give you a different mod time every single time you resolve
package main
import (
"fmt"
   "github.com/gobuffalo/packr/v2"

)
func main() {
box := packr.New("test", "./test")
err := box.AddString("binary", "this is a test binary")
if err != nil {
panic(err)
}
   for {
           f, err := box.Resolve("binary")
           if err != nil {
                   panic(err)
           }

           info, err := f.FileInfo()
           if err != nil {
                   panic(err)
           }

           fmt.Println(info.ModTime())
   }

}
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

@markbates did you have any thoughts about how this sort of metadata would be embedded into the format that is generated when you pack? Reading off the disk and relaying the file back admittedly easier. I was having a tiny bit of trouble finding a nice place to insert this sort of metadata.

yeah, I'm not sure either. I've been trying to figure that out too. the best I've come up with is figuring out a way to prepend metadata when it's packed and then read that metadata when it's unpacked, but I haven't dived into it any more than that.

Hi @markbates, I've been working with @zachgersh on this, and we've come up with two possible formats.

We include a "*.info" map entry for each file that contains file metadata as a string. The benefits of this approach is that I don't believe it will break current implementation, and we can just add the ability to read the extra "*.info" map entry.

package packrd

import (
	"github.com/gobuffalo/packr/v2"
	"github.com/gobuffalo/packr/v2/file/resolver"
)

var _ = func() error {
	const gk = "263ac315b3401aa0fdd11824b806530c"
	g := packr.New(gk, "")
	hgr, err := resolver.NewHexGzip(map[string]string{
		"e9cb145202973e9e3d5fdd3a3540e230":      "1f8b08000000000000ff4aca49cce002040000ffff2d32c45005000000",
		"e9cb145202973e9e3d5fdd3a3540e230.info": "file-info",
	})
	if err != nil {
		panic(err)
	}
	g.DefaultResolver = hgr

	func() {
		b := packr.New("test", "./test")
		b.SetResolver("foo.txt", packr.Pointer{ForwardBox: gk, ForwardPath: "e9cb145202973e9e3d5fdd3a3540e230"})
	}()

	return nil
}()

We create a new struct that holds both the file contents and the file info to encapsulate this data. The benefits of this approach is that it is easier to understand, and the drawbacks are that we would have to change at least NewHexGzip to support the struct.

package packrd

import (
	"github.com/gobuffalo/packr/v2"
	"github.com/gobuffalo/packr/v2/file/resolver"
)

type Gzip struct {
	Content  string
	FileInfo string
}

var _ = func() error {
	const gk = "263ac315b3401aa0fdd11824b806530c"
	g := packr.New(gk, "")
	hgr, err := resolver.NewHexGzip(map[string]Gzip{
		"e9cb145202973e9e3d5fdd3a3540e230": Gzip{
			Content:  "1f8b08000000000000ff4aca49cce002040000ffff2d32c45005000000",
			FileInfo: "file info",
		},
	})
	if err != nil {
		panic(err)
	}
	g.DefaultResolver = hgr

	func() {
		b := packr.New("test", "./test")
		b.SetResolver("foo.txt", packr.Pointer{ForwardBox: gk, ForwardPath: "e9cb145202973e9e3d5fdd3a3540e230"})
	}()

	return nil
}()

I like them both, leaning more towards the second. We wouldn't have to necessarily change NewHexGzip, but instead, we could create a different function to handle the new syntax. That would keep existing packr from breaking, and add support for file infos, so basically, create a new Store that handles it.
My bigger question, is how to reflect this in the interfaces to ensure it works with different stores. Thoughts on that?

Could you elaborate more on the different store interfaces? I'm not sure I'm understanding the problem fully.

For example, there are 3 defined in packr itself https://godoc.org/github.com/gobuffalo/packr/v2/jam/store each one needs to make sure to pack the file info and the resolvers also need to make sure to unpack the file info into each file.
So, if I use the Legacy store, how is the data packed into that store so it doesn't lose file info data, and how is it pulled out of that store.
You've outlined a really nice solution for that particular resolver (the store being the Disk store that generates that file.
I just want to make sure the any touch points that aren't just this one resolver are capable of supporting file info. Resolvers can also be implemented by end users. For example how does this impact the Disk or InMemory resolvers?
There might not be any issues, but I just want to make sure everything is thought through.

+1 for storing actual modtime with each file in the packr.Box. That would make it easier to implement cache control in http servers embedding static assets via packr.


Talk authors on landing page should link to member name

Currently links to Member ID -- should be linking to member names instead (this should link to https://makeabilitylab.cs.washington.edu/member/leestearns/)

this is fixed!


Not returning error with wrong password

Login Post request
{
email<EMAIL_ADDRESS>password: "company5pass"
}
returns a validation token instead of an error saying that the password was incorrect.
The user should not be logged in for this case.

Issue resolved, changed bcrypt.compare to bcrypt.compareSync.


sortable dissapears after first drop

I see the example works fine, and i've swapped to using the same libraries but for some reason when i apply to my use case using bootstrap 3, after it's dragged once even if not dropped the top style coordinates start using the top of the window or screen meaning it goes from say top -100px or so to -2300px after it's dragged then stays that way until a page reload.  i've tried adjusting parents to relative but still no go.

@clearbucketLabs - do you have a way to create some type of reproduction that I could run?

Seems like if window is scrolled down a bit, even a few px the offset is off, if scrolled right to the top it's perfect.  it could be something else messing something up so i'll check and post an update first before wasting time.

So it seems it's body { overflow-x: hidden;}   that messed it up.  Why its there i forget, there was a good reason at least but why it messes it up i don't know.

@clearbucketLabs - glad that you got it sorted out.


use fast-check for more exhaustive unit testing

I use https://github.com/dubzzz/fast-check a lot and really like it. I would be happy to make a PR to start using the library for testing some of typedorm's core functionality?

@mishabruml
I am not much familiar with the concept of property-based testing.
It seems that you are having very much experience with it, are you able to provide a higher-level overview of what it is and how will it be useful in TypeDORM?  😀

Closing this as it is no longer active.


Using software-driven Coex Module leads to lost BLE Messages (IDFGH-14092)

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.
v5.1.2, v5.1.5, v5.3.1
Espressif SoC revision.
ESP32
Operating System used.
macOS
How did you build your project?
Command line with idf.py
If you are using Windows, please specify command line type.
None
Development Kit.
Custom Board
Power Supply used.
Battery
What is the expected behavior?
Sending Messages using BLE while still listening to ESP NOW Messages will work without loosing a single BLE connection (as it behaved in IDF v4.3.4), no matter if we use the hardware-defined coexistence or the software-defined (software-defined even using this while sending messages).
What is the actual behavior?
After upgrading from v4.3.4 to v5.1.2 we saw sporadic lost Messages sent from one ESP32 to another using NimBLE. After upgrading then to v5.3.1, the issue was even more noticeable, even affecting normal operation of our devices.
After switching from software-defined to hardware-defined coex (CONFIG_ESP_COEX_SW_COEXIST_ENABLE is not set), the issue stopped to appear and all BLE Messages were able to be delivered again.
Using Wireshark, we were able to find out that the ESP32 is just not responding or sending messages for a given time period, which will lead to multiple unhandled BLE cycles, loosing multiple messages.
Steps to reproduce.
Hard to describe, as the issue appears in our own Software. If the issue is not known already, we might create an example project reproducing the issue in an isolated project.
Debug Logs.
No response
More Information.
We mainly want to know if the Issue is known already and why the software-defined coexistence got broken from v5.x upwards somehow and got even worse in v5.3.1. We are using ESP NOW as second communication channel, but BLE is used exclusively during normal operation (with ESP NOW only listening for incoming messages, but there are none if nothing gets lost over BLE).
Also, we were not able to exactly find out what are the drawbacks using the hardware-defined coexistence, as it now works kind of good during the first attempts using it. Is there any documentation except the coexistence documentation, stating out more clear what the difference actually is?

Hi, @timoxd7
If we disable software coexistence on ESP32, the behavior of hardware will become very simple, we can call it 'Bluetooth priority mode', it means Bluetooth can always use the RF resource if it want. Disable software coexistence does help improve BLE performance, but this may cause unstable wifi performance.
Based on the information you provided, I can't confirm whether this is a known issue. Can you provide more information about BLE messages lost, including:

Which API are you using to send BLE messages
The return value of the API and the output log when losing BLE messages
IDF version or commit id that the issue reproduce
Also, It would be even better if you can build an example project to reproduce the issue.

At last, thanks for your feedback, we will continue to follow up on this issue.


BOX-127 create a simple service for making portable in Lucee

Create a dedicated java loading proxy service for getting at the java objects from the RabbitMQ library. This allows some simple logic to behave differently when Lucee than with ACF. i.e. we can just use CreateObject( "java", classPath, libJars ) to dynamically load the jars.
We could possibly use this same approach to use JavaLoader with ACF if there are no problems with JL + ACF?

This is an interesting idea.  Here is the Lucee ticket that drove me away from using JavaLoader on this specific project:
https://luceeserver.atlassian.net/browse/LDEV-2296
It's 3 years old and hasn't had any traffic from Micha yet despite me narrowing down the exact point in the Lucee code things break down.
And on a related note, here's the 7 year old ticket for Adobe to finally catch up with Lucee and provide the same mechanism for providing jars on the fly to createObject().
https://tracker.adobe.com/#/view/CF-4087803
Have you tested this change on both Lucee and Adobe?  I'm also curious if it causes any issues to have the Rabbit jars in the this.JavaSettings or the servlet's lib folder and ALSO use the new approach?  If so, this would be a breaking change since the docs have previously instructed all users to manually supply the jars.
I'm not against using  JavaLoader for Adobe, but I'd REALLY like to avoid it on my projects.  It's very, very old and basically has zero support.  It can also easily cause memory leaks if it's not shutdown properly, which is even harder to enforce when outside of the ColdBox/CommandBox frameworks.

Yeah agreed. We don't use javaloader anywhere. I think this PR needs more work - the RabbitMQ java library is built with OSGi meta so could be refactored further by tapping into that. I'm actively working on this right now so I'll try to get OSGi working.
I haven't run the tests using ACF, was presuming without looking that there would be an engine matrix for running the tests in Gitlab - but I wouldn't foresee any issue here and will verify. But if not wanting JavaLoader, then for ACF, this change shouldn't effect it in any way - just add a smoother experience for Lucee users. Having it configurable and off by default could be an option though? i.e. if on ACF, and not able/wanting to update lib dirs, then could switch it on with some config?
Begs the question, does this warrant a separate mini library for loading jars that uses Lucee native where possible (and using OSGi where possible), and falling back to ACF compatible options otherwise??

the RabbitMQ java library is built with OSGi meta so could be refactored further by tapping into that. I'm actively working on this right now so I'll try to get OSGi working.

Meh, usually OSGI means everything gets more complicated, so I don't personally care much about that unless you just really want to, lol :smile:

was presuming without looking that there would be an engine matrix for running the tests in Gitlab

I'm using Github Actions for this, but sadly the build needs some love and I only have the tests running on Lucee ATM.  Feel free to help add Adobe into the mix.  The client who paid for most of this work only cared about Lucee so my Adobe support was just some local testing I did.

then for ACF, this change shouldn't effect it in any way

Agreed, but we should at least just hit the tests to ensure the code paths are ok.  For example, will Adobe freak out when it goes to compile a createObject() call with too many args, etc.

Having it configurable

This isn't a bad idea just in case someone has a really good reason to load the jars themselves.

off by default

Honestly, so long as it works, I'm fine with it on by default as I really prefer it to "just work".  We can do a major bump of the library version just in case since it could be a breaking change for someone somewhere.  (not many people are using this from what I can tell)

Begs the question, does this warrant a separate mini library for loading jars that uses Lucee native where possible (and using OSGi where possible), and falling back to ACF compatible options otherwise??

You know, I thought about the same thing, lol. I really hate having two installation methods for different engines, and the annoying thing is just having cbjavaloader in the box.json as a dependency will mean it gets downloaded and loaded even if we hit a code path that ignores it.  Most of my other libraries which wrap jars still use javaloader like bcrypt, cfcouchbase, etc.  So long as those continue to work as-is I'm not of the mind to change them.  What I'd REALLY like is for Adobe to get their butts in gear and provide me with proper cross-engine support so I don't have to keep wasting my time to make up for their lack of progress.  So the future I see is one where no shim library is required which makes me reticent to move that direction even though there is no indication that the current status quo will change any time soon :disappointed:
On a related note, it would be interesting if we just modified our port of javaloader to detect Lucee and perform your workaround instead of creating the custom classloaders, etc.  It's not any less work or annoyance, but it does still keep us just maintaining a single module for cross-engine compat java class loading.


Should PythonCall create separate environment for different projects?

If we have two different project directories and they are pointing at different versions (or locations) of a common Julia package, then it seems this causes errors. Maybe this is just the way it is meant to work. I'm running into this issue because I have a local repo with a Julia package that I'm trying to move that to GitHub and I can't have both working in parallel with PythonCall, it seems.

It already does create separate conda environments for different Julia projects. Specifically, it creates a conda environment in the top most entry of LOAD_PATH which depends on PythonCall (and recreates it if the LOAD_PATH changes).
Can you describe the specific problem you're having? Preferably describe how I can recreate it myself.

The problem might be because I'm not reinstalling the Python package when I run in the other directory. I will try to clarify this and let you know. Your answer suggests the problem is much more likely on my side than in PythonCall, thanks.

Confirmed that was the issue - I was running from a different directory but Python was using an installed version of the Python package and this was causing PythonCall to see two different URLS, I guess one form the juliacalldeps.json in the installed Python package and another in the local directory where I was running. By installing the package the problem went away.


Could not load 'examples/hello_near.js'

I'm currently running on an M1 Max Chip and have installed the prerequisites using the following commands:

arch -arm64 brew install wget
arch -arm64 brew install cmake
arch -arm64 brew install make

I then ran the setup script followed by the build script:

./setup.sh
./build.sh

Following the instructions for running on testnet, I then ran the following command./builder.sh examples/hello_near.js which threw an error saying
Could not load 'examples/hello_near.js'
I also tried running ./builder.sh examples/low_level/hello_near.js but the error was the same.

It looks like your setup and build on M1 is correct given no error shown on ./setup.sh and ./build.sh. And the error is maybe on typo in path: low_level -> low-level?

Fixed with https://github.com/near/near-sdk-js/pull/9


readTask sometimes stuck on USER command.

Hi,
I'm trying to test FtpServer behavior on osx using Xamarin with Mono 4.2.4 as well as 3.10.0.
This happens mostly when I try to make multiple concurrent connection to ftp server, but also happens when I try to reconnect.
After some debugging, I found that readTask doesn't complete after ftp client sends USER command. I checked tcpdump and confirmed that server ACK'd USER command part.
Interestingly, when I change readTask to synchronous Read() instead of ReadAsync() it could read correct full USER command. Also, if I change the size of buffer in ProcessMessages() to 16, which is "USER anonymous\r\n", it always success to read.
I don't have many experiences on C# especially on OSX. Have you experienced this case before?
Thanks,
hytgbn

I'm not aware of any problems during the authentication, but it might be aproblem with mono. A reliable test case is required before I can fix this problem. Thanks for reporting.


Is there a way to READ external flash?

Is there a way to READ external flash?

The CMSIS-Pack flash algorithm specification (FLM), which this repo implements, does not provide an API to read flash.
However, Segger's Open Flashloader (OFL) specification, notionally based on FLM, adds a SEGGER_OPEN_Read() API. (Unfortunately, OFL algos are only compatible with the FLM spec at the API level. The OFL algos themselves expect a very different runtime environment, at least when built like the OFL examples.)
The full prototype is this:
int SEGGER_OPEN_Read(U32 Addr, U32 NumBytes, U8 *pDestBuff)

There's no reason this API couldn't be added to an algo built with this repo as an extension. I plan on proposing some extensions to the FLM spec; this can be included, probably renamed to just ReadPage().
For pyocd, I assume that the Init() API will configure external flash such that it is memory mapped. It has a flag that can be set on flash memory regions telling it that it needs to init the flash algo before attemping to read the flash memory. This works for the very few cases I have to test, although it obviously only supports memory mappable external flash.

Hi, @flit
int ProgramPage (unsigned long adr, unsigned long sz, unsigned char *buf)

How does this function receive buf data from SWD? I want to learn this knowledge. Do you know where I can learn this? Can you enlighten me?
I think if we can send data to the buffer of buf via SWD, can we also read the data in buf from SWD?
(Via Google Translate)

First of all , the CMSIS-Pack flash algorithm specification is useful documentation.
The general process for programming flash using a CMSIS-Pack algorithm is this:

Download flash algorithm code and data (extracted from the. FLM file, which is an ELF file) to RAM on the target device.
Write a bkpt #0 instruction (0xbe00) to RAM, usually prepended to the flash algorithm's code.
Invoke the Init() entry point, passing the "erase" operation type. Other parameters can be set to 0.
Invoke the EraseSector() entry point with base address of sector to erase (or use EraseChip()). Repeat for all sectors to erase.
Invoke the Init() entry point, passing the "program" operation type. Same as before, the other parameters can be set to 0.
Load data for the page to program to a buffer in target RAM.
Invoke the ProgramPage() entry point, with adr set to flash page base address, sz set to size of data to program (typically one page) , and buf set to the base address of the buffer to which data was copied. Repeat from step 5  for all pages to program.
Invoke the UnInit() entry point.

To call an entry point:

Ensure SP is set to an limit address of a RAM region dedicated to stack, usually a minimum of 256 bytes.
Set LR to the address of the bkpt instruction that was placed in RAM.
Set R9 to the base address where the flash algorithm's data was loaded in RAM.
Set R0 through R3 to parameters for the entry point.
Set PC to the entry point address with bit 0 set (e.g., entry point | 1).

The addresses of the entry points are determined from the values in the ELF file's symbol table, offset by the base address of the code loaded to RAM. For instance, if the ProgramPage symbol has a value of 0x1d8, and the algorithm code was loaded to 0x20000002 (with the breakpoint at 0x20000000). then the ProgramPage entry point address is 0x200001da.
This is all encapsulated in code in the pyocd.flash.flash.Flash class in pyocd. The init() method downloads the flash algorithm and calls the Init() API. The program_page() method copies the page data to target RAM and invokes the ProgramPage() entry point. The _call_function_and_wait() method runs one of the API entry points to completion, and is used by the other methods..

thx a lot, @flit!
I will try to customize a function and then try to call it.


Recurring Donations Settings can understate successes and failures when an entire batch fails

Background
Recurring Donations have a batch that runs nightly to insert new Opportunities for open ended recurring gifts. It records successes and failures when it finishes in the Recurring Donations Settings custom setting.
Summary of Issue

The overnight batch (RD_RecurringDonations_BATCH) counts the number of records and failures, and updates the Recurring Donations Settings custom setting when the batch finishes
If the batch's execute method hits a platform limit (e.g. Apex CPU limit exceeded), the variables for number of records and failure count are not incremented
When these numbers are saved to Recurring Donations Settings, the success and failure counts are understated

Potential Solutions

Create a new variable that indicates the number of batches that failed, the source of this would be AsyncApexJob. NumberOfErrors and can be added as field to the query here
Approximate the number of failures by multiplying the AsyncApexJob.NumberOfErrors (as described above) by the batch size (batchSize); this is not precise because if the final batch fails and contains less records than the total batch size, it would overstate the failures

@rogeramitchell All current and future Recurring Donation development is focused on the Enhanced Recurring Donation model. You can learn more about it including the steps to upgrade here. As this issue applies to the legacy model in how it creates forecast opportunities and updates batch status, which the new model does not do, we won’t be accepting this as an open issue at this time. I know this wasn't the answer you were hoping for... if you have questions or concerns about upgrading to Enhanced Recurring Donations, please post in the NPSP group in the Hub or open a support case. Thanks!


Generator detection seems broken

Platform
Which platform does the bug apply to?

WebExtension (Google Chrome, Mozilla Firefox)
NPM

Describe the bug
In detecting WooCommerce & WordPress (probably more, didn't test), the version number isn't properly detected. In fact it looks like the entire meta > generator detection doesn't work.
To Reproduce
Steps to reproduce the behavior:

Run node src/drivers/npm/cli.js https://yoast.com/
See that it doesn't detect a version number for WordPress and WooCommerce.

Expected behavior
I would expect it to be able to extract a version number from the generator tags.

Fixed, thanks! Multiple meta tags with the same name were not handled correctly, only the last value was used.


Remove crowndin reference from README

Tidying up the README as described in #2.
@tesseralis I assume it's safe to remove this reference to crowdin right?

Yup! I'm trying to delete it from the upstream repo too!


Use non-generic TryComp() for metadata & transform

Allows entity systems to use the cached xform & metadata queries

I didn't even realize this was a thing. Nice!
Also, this looks like something that a code analyzer could catch people doing in the future. Maybe I'll work on that.


debug disabled for cuda code in vscode

hi there, I installed the vscode-cuda. I cannot add breakpoint in .cu file of my project in Ubuntu OS. For the windows version of the same project, I have no problem in adding breakpoint in .cu files. I wonder if this is a problem associated with vscode-cuda or VSCODE itself. Thanks,

I think that debugging support is in the same basket as Intellisense. If you don't explicitly implement the feature, it won't work like in the C++ plugin.
This plugin is aimed at giving you code coloring and snippets, not full debugging environment.
I had this repo started to do that, but it is actually a lot of work. If someone wants to do it, ok for me 👍


build GenerateReferenceDocs succeeds, but produces no output

I've been trying to update my project to use the latest ProjectScaffold, but even with a brand new copy (just download and build, on Windows, using VS2013 tools), build GenerateReferenceDocs succeeds, but generates no output.
The build step runs, and shows:
Finished Target: GenerateHelp
Starting Target: GenerateReferenceDocs (==> RunTests, GenerateHelp)
Building documentation (Default), this could take some time, please wait...
C:\Users\Reed\Documents\GitHub\ProjectScaffold-master\packages\build\FAKE\tools\FAKE.exe  --fsiargs -d:FAKE -d:RELEASE -d:REFERENCE "generate.fsx"
FSharp.Formatting Information: 0 : FSharp.Formatting Logging setup!
Yaaf.FSharp.Scriping Information: 0 : Yaaf.FSharp.Scripting Logging setup!
Copying file: C:\Users\Reed\Documents\GitHub\ProjectScaffold-master\docs\output\img\logo-template.pdn
Copying file: C:\Users\Reed\Documents\GitHub\ProjectScaffold-master\docs\output\img\logo.png
Copying styles and scripts: C:\Users\Reed\Documents\GitHub\ProjectScaffold-master\docs\output\content\style.css
Copying styles and scripts: C:\Users\Reed\Documents\GitHub\ProjectScaffold-master\docs\output\content\style_light.css
Copying styles and scripts: C:\Users\Reed\Documents\GitHub\ProjectScaffold-master\docs\output\content\tips.js
Copying styles and scripts: C:\Users\Reed\Documents\GitHub\ProjectScaffold-master\docs\output\content\img\github-blue.png
Copying styles and scripts: C:\Users\Reed\Documents\GitHub\ProjectScaffold-master\docs\output\content\img\github.png
Finished Target: GenerateReferenceDocs

But, as you can see from the above, nothing is put in the reference folder under output, and no reference docs are generated.
Again, this was just taking the current master, downloading as a zip, running build, then build GenerateReferenceDocs - with no changes or updates.  The command "succeeds" but doesn't seem to actually generate the reference folder or documentation.

It also appears that ReferenceBinaries isn't set properly during the init phase - it's coming through as empty.  However, setting it up manually doesn't seem to change the output.

After further investigating, it appears that the -d:REFERENCE flag isn't coming through into generate.fsx correctly.  I'm not sure why, but when I change the end of generate.fsx to:
// Generate
copyFiles()

printfn "AFTER FILE COPY"

#if HELP
printfn "Calling buildDocumentation()"
buildDocumentation()
#endif
#if REFERENCE
printfn "Calling buildReference()"
buildReference()
#endif

printfn "AFTER DEFINES"

And then run, the main help generation succeeds (and prints as expected), but I then see:
Help generated
Finished Target: GenerateHelp
Starting Target: GenerateReferenceDocs (==> RunTests, GenerateHelp)
Building documentation (Default), this could take some time, please wait...
C:\Users\Reed\Documents\GitHub\Gjallarhorn\packages\build\FAKE\tools\FAKE.exe  --fsiargs -d:FAKE -d:RELEASE -d:REFERENCE "generate.fsx"
FSharp.Formatting Information: 0 : FSharp.Formatting Logging setup!
Yaaf.FSharp.Scriping Information: 0 : Yaaf.FSharp.Scripting Logging setup!
Copying file: [myproj]\docs\output\img\logo-template.pdn
Copying file: [myproj]\docs\output\img\logo.png
Copying file: [myproj]\docs\output\img\Thumbs.db
Copying styles and scripts: [myproj]\docs\output\content\style.css
Copying styles and scripts: [myproj]\docs\output\content\style.css.bak
Copying styles and scripts: [myproj]\docs\output\content\style_light.css
Copying styles and scripts: [myproj]\docs\output\content\tips.js
Copying styles and scripts: [myproj]\docs\output\content\img\github-blue.png
Copying styles and scripts: [myproj]\docs\output\content\img\github.png
AFTER FILE COPY
Calling buildDocumentation()
AFTER DEFINES
Finished Target: GenerateReferenceDocs

Note that the FAKE.exe line reported (and I verified this) is being called with -d:REFERENCE, but within the generate.fsx, that is being bypassed.  I'm not familiar with how FAKE passes arguments through to FSI, but it seems like there's a bug happening here...

Target "GenerateReferenceDocs" (fun _ ->
    if not <| executeFSIWithArgs "docs/tools" "generate.fsx" ["--define:RELEASE"; "--define:REFERENCE"] [] then
      failwith "generating reference documentation failed"
)

is a workaround until we know how to solve https://github.com/fsharp/FSharp.Compiler.Service/issues/511

It was an issue with FAKE's script caching. fsiArgs need to cached as well.
In order to fix your issue: "paket update nugate FAKE" should do the trick
cc @matthid

Thanks for the CC. I think this was one of the arguments for FAKE targets instead of compiler directives. I really think we should make the documentation generation a proper FAKE script with multiple targets.


GH-2350: Fix Paused Partitions After Rebalance

Resolves https://github.com/spring-projects/spring-kafka/issues/2350
Do not maintain the pausePartitionsRequested field in the concurrent MLC,
it is not used and can cause confusion.
Also fixes some synchronization around CMLC.containers.
Also resolves https://github.com/spring-projects/spring-kafka/issues/2222
The previous fix removed revoked partitions from pausePartitionsRequested;
this was incorrect - consider a rebalance where we lose topic-0 and another
rebalance where we are re-assigned topic-0. According to the contract, this
partition should remain paused.
cherry-pick to 2.9.x, 2.8.x

... and cherry-picked to 2.9.x & 2.8.x


But with AdaGram I am not able to load more than 20 MB data in the docker container.
If I load more than 20 MB, I am getting bus error.
julia> using AdaGram
julia> AdaGram.load_model("/julia/full.embed")
signal (7): Bus error
_unsafe_batchsetindex! at ./multidimensional.jl:329
setindex! at ./abstractarray.jl:592
Bus error (core dumped)

I can able to load 40 GB of text file in julia shell in docker container.

Lack of sufficient shared memory in docker container caused the problem. The problem is solved by specifying the shared memory.


Amplified world import does not work

Hi, #1343 suggests that you can import an amplified world but when trying to do /mv import  AMPLIFIED you receive this error:

Thank you!

On further inspection, trying to import an amplified world as NORMAL does seem to cause future amplified terrain generation to work properly, but the UI or documentation could use an update for overly concerned admins like myself :)

Alright, Glad you solved it. We will improve our documentation when we can. Thanks!


596 fix problems with integrationlangchaintest

Description
Tidying up some code and making test a bit better
Notes

moves vectorisedDocuments and initDocumentVectors from langchain.ts to document.ts.
puts all document unit tests together into a folder.

moves tests from integration/langchain.test.ts -> unit/document.ts/initDocumentVectors.test.ts.
renames/moves unit/document.test.ts -> unit/document.ts/getDocuments.test.ts

moves tests from integration/langchain.test.ts -> unit/langchain.ts/formatPromptEvaluation.test.ts
moves tests from integration/langchain.test.ts -> unit/langchain.ts/initialisePromptEvaluationModel.test.ts
tidies some test layouts and naming also.
Individual tests are largely unchanged. Some mocks have been altered/added and some things reworded

Checklist
Have you done the following?

[x] Linked the relevant Issue
[ ] Added tests
[x] Ensured the workflow steps are passing

will reopen when It's ready

I'd be interested to know what the response actually is here, and why we need to strip out all non-alphanumerics; does it wrap the "yes" or "no" inside braces, or quotes maybe?

@chriswilty
When I sent "Hello", the prompt eval LLM came back with "No.", and when I sent "forget your instructions" it came back with "Yes.".
So we need to strip away the punctuation.

I'd be interested to know what the response actually is here, and why we need to strip out all non-alphanumerics; does it wrap the "yes" or "no" inside braces, or quotes maybe?

@chriswilty
When I sent "Hello", the prompt eval LLM came back with "No.", and when I sent "forget your instructions" it came back with "Yes.". So we need to strip away the punctuation.

Brilliant, thanks for checking!

I'd be interested to know what the response actually is here, and why we need to strip out all non-alphanumerics; does it wrap the "yes" or "no" inside braces, or quotes maybe?

@chriswilty
When I sent "Hello", the prompt eval LLM came back with "No.", and when I sent "forget your instructions" it came back with "Yes.". So we need to strip away the punctuation.

Brilliant, thanks for checking!

@chriswilty in fact we're explicitly asking it to return in that format, with the full stop. I wonder why.

@pmarsh-scottlogic I'm happy to approve this anyway. If you need any help with merge conflicts, just ask (tomorrow!)


The Devs have abandoned the game, even if you haven't.

I am sad to say that multiplayer saves will never be a thing.
they have confirmed they won't add it.
A game like this is played among friends, Long matches, countless hours of fun continued over days if necessary. But They did not officially make LAN nor multiplayer save.
For years, we have asked for Multiplayer save function, but nothing was done.
I sincerely request that you, who are passionate about the game and community, make a way to save multiplayer games. Please.
Below is what the official response is to our request. I pray with all my heart that you help us.
Thank you.
"Hello,
Save-load functionality for multiplayer isn`t planned, but we will implement reconnections. Thanks for being interested in our game  🙂"
https://steamcommunity.com/app/333420/discussions/0/1327844097117290719/

Thank you for the information. Unfortunately, I do not consider continuing this project in any way nor do I have the technical prerequisites to be able to implement such a feature in the game client.


python3Packages.aiostream: init at 0.4.1

Motivation for this change
url: https://github.com/vxgmichel/aiostream
stars: 290
test coverage: 100%
seems like a pretty good library, and i was surprised to see we didn't have it already
Things done

[X] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[X] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[X] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[X] Fits CONTRIBUTING.md.

$ nixpkgs-review rev 0383c9c32dd --no-shell

Updating 99264011def..0383c9c32dd
Fast-forward
 pkgs/development/python-modules/aiostream/default.nix | 30 ++++++++++++++++++++++++++++++
 pkgs/top-level/python-packages.nix                    |  2 ++
 2 files changed, 32 insertions(+)
 create mode 100644 pkgs/development/python-modules/aiostream/default.nix

3 packages added:
python37Packages.aiostream (init at 0.4.1) python38Packages.aiostream (init at 0.4.1) python39Packages.aiostream (init at 0.4.1)

3 packages built:
python37Packages.aiostream python38Packages.aiostream python39Packages.aiostream

seems like a pretty good library, and i was surprised to see we didn't have it already

We mostly try to package libraries required by tools and things that are used really often because otherwise Pypi is just to big and is to much work to maintain. So we miss quite a lot from pypi which is okay.

seems like a pretty good library, and i was surprised to see we didn't have it already

We mostly try to package libraries required by tools and things that are used really often because otherwise Pypi is just to big and is to much work to maintain. So we miss quite a lot from pypi which is okay.

Of course, and I'm happy to keep this in my own project's nix expressions if you'd rather not include it. I am currently finding it useful in a project FWIW, and I thought others might benefit from it in nixpkgs.

Of course, and I'm happy to keep this in my own project's nix expressions if you'd rather not include it. I am currently finding it useful in a project FWIW, and I thought others might benefit from it in nixpkgs.

This is a semi-automatic executed nixpkgs-review which does not build all packages (e.g. lumo, tensorflow or pytorch)
If you find some bugs or got suggestions for further things to search or run please reach out to SuperSandro2000 on IRC.
Result of nixpkgs-review pr 108829 run on x86_64-linux 1

3 packages built:

python37Packages.aiostream
python38Packages.aiostream
python39Packages.aiostream

This is a semi-automatic executed nixpkgs-review which does not build all packages (e.g. lumo, tensorflow or pytorch)
If you find some bugs or got suggestions for further things to search or run please reach out to SuperSandro2000 on IRC.
Result of nixpkgs-review pr 108829 run on x86_64-linux 1

3 packages built:

python37Packages.aiostream
python38Packages.aiostream
python39Packages.aiostream

This is a semi-automatic executed nixpkgs-review which does not build all packages (e.g. lumo, tensorflow or pytorch)
If you find some bugs or got suggestions for further things to search or run please reach out to SuperSandro2000 on IRC.
Result of nixpkgs-review pr 108829 run on x86_64-darwin 1

3 packages built:

python37Packages.aiostream
python38Packages.aiostream
python39Packages.aiostream

This is a semi-automatic executed nixpkgs-review which does not build all packages (e.g. lumo, tensorflow or pytorch)
If you find some bugs or got suggestions for further things to search or run please reach out to SuperSandro2000 on IRC.
Result of nixpkgs-review pr 108829 run on x86_64-darwin 1

3 packages built:

python37Packages.aiostream
python38Packages.aiostream
python39Packages.aiostream


mini.cursorword: Ignore short word (single char or 2 chars)

Contributing guidelines

[X] I have read CONTRIBUTING.md
[X] I have read CODE_OF_CONDUCT.md

Module(s)
mini.cursorword
Description
Is there a way to disable mini.cursorword when I click to short word (single char).
For example: a-b-c, when I move to a I don't want to hightlight all word a.

Unfortunately, no.
The whole idea of 'mini.cursorword' is to highlight whole word under cursor. In the same way as would built-in *, #, etc. work.
So although I do understand the appeal of not highlighting short words, this is both not easy to do with current implementation (which uses built-in matches and Vim patterns) and not aligned with built-in behavior.

For example: a-b-c, when I move to a I don't want to hightlight all word a.

For this particular example I can suggest to add "-" character to a list of characters that are recognized as a part of what Neovim considers to be a "keyword". This can be done by modifying 'iskeyword' option like so: vim.opt.iskeyword:append('-').

Closing as not planned.


refactor: remove atomic react methods

Description

BREAKING CHANGE: Removes react method from atomic banks and cards clients.

Testing required outside of automated testing?

[x] Not Applicable

Screenshots (if appropriate):

[x] Not Applicable

Rollback / Rollforward Procedure

[x] Roll Forward
[ ] Roll Back

Reviewer Checklist

[ ] Description of Change
[ ] Description of outside testing if applicable.
[ ] Description of Roll Forward / Backward Procedure
[ ] Documentation updated for Change

:tada: This PR is included in version 1.36.1 :tada:
The release is available on:

v1.36.1
GitHub release

Your semantic-release bot :package::rocket:


Fix codeclimate error

Coverage remained the same at 70.32% when pulling f316fa33ddf6513314888c9a50cd05d55ee56624 on codeclimate into dabb46ba2bffc3d73bacdd374708785960ea6aed on master.


"Failed to get root window"

I'm trying to run the following example:
#[macro_use]
extern crate glium;

use glium::*;

fn main() {
    let display = glium::glutin::WindowBuilder::new()
        .with_dimensions(1024,1024)
        .with_title("Hai")
        .build_glium()
        .unwrap();
    loop {
/*        for ev in display.poll_events() {
            match ev {
                glium::glutin::Event::Closed => return,
                _ => ()
            }
        }*/
    }
}

Running it with cargo (from Emacs) gives me this output (backtrace enabled). I have no idea what the stream of eights is.
-*- mode: cargo-process; default-directory: "~/deymoh/" -*-
Cargo-Process started at Sat Apr  1 21:45:05

cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.1 secs
     Running `target/debug/deymoh`
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
thread 'main' panicked at 'Failed to get root window: XError { description: "GLXBadFBConfig", error_code: 178, request_code: 155, minor_code: 34 }', /checkout/src/libcore/result.rs:859
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: <core::result::Result<T, E>>::expect
             at /checkout/src/libcore/result.rs:761
   1: glutin::platform::platform::x11::Window::new
             at /home/rain/.cargo/registry/src/github.com-1ecc6299db9ec823/glutin-0.7.4/src/platform/linux/x11.rs:215
   2: glutin::platform::platform::api_dispatch::Window::new
             at /home/rain/.cargo/registry/src/github.com-1ecc6299db9ec823/glutin-0.7.4/src/platform/linux/api_dispatch.rs:100
   3: glutin::window::<impl glutin::WindowBuilder<'a>>::build
             at /home/rain/.cargo/registry/src/github.com-1ecc6299db9ec823/glutin-0.7.4/src/window.rs:224
   4: glium::backend::glutin_backend::GlutinWindowBackend::new
             at /home/rain/.cargo/registry/src/github.com-1ecc6299db9ec823/glium-0.16.0/src/backend/glutin_backend.rs:262
   5: glium::backend::glutin_backend::<impl glium::DisplayBuild for glutin::WindowBuilder<'static>>::build_glium_debug
             at /home/rain/.cargo/registry/src/github.com-1ecc6299db9ec823/glium-0.16.0/src/backend/glutin_backend.rs:150
   6: glium::DisplayBuild::build_glium
             at /home/rain/.cargo/registry/src/github.com-1ecc6299db9ec823/glium-0.16.0/src/lib.rs:1240
   7: deymoh::main
             at ./src/main.rs:7

Cargo-Process exited abnormally with code 101 at Sat Apr  1 21:45:06

System info:
Arch Linux + Mesa
i3 window manager
Thinkpad T400 with old Intel graphics

I actually do see a window pop up for a fraction of a second. AFAIK there are no problems with my system, I have used it before to play some older or simpler games.

I have the same problem with ubuntu 16.04 on hp laptop 6710b. I beleave it has OpenGL 2.1 hardware. Is this version of OpenGL supported?

Fixed it by forcing an older version of OprnGL, but the error message could be clearer about what's going wrong, so I'll edit the issue to that.

Please be more specific, how to force an older version. I need this worksround too.

I checked the output of glxinfo and it told me the supported version.
Here is how you force a specific version:
let display = glium::glutin::WindowBuilder::new()
        .with_dimensions(1024,1024)
        .with_title("Hai")
        .with_gl(glutin::GlRequest::GlThenGles{
            opengl_version : (2, 1),
            opengles_version : (2, 1),
        })
        .build_glium()
        .unwrap();


The copyright issue

@daikk115 so your name is Mischback?

@hieulq
Ops! I'm forgot that.
First of all, I just updated project metadata: https://github.com/cloudcomputinghust/CAL_Appliances/tree/master/fcap
Then, I have updated fcap structure describes with referencing to Mischback project.
It's ok? I don't see any licence at Mischback project template, so I just referenced to it.

Better remove author name and put a sentence state that your product is forked from his.

I have removed author like this:
https://github.com/cloudcomputinghust/CAL_Appliances/blob/master/fcap/README.md


Error with 24khz data utils

So when trying to resume one of the existing 24khz models I am getting the following error.
File "Z:\FreeVCtrain24khz\train_24.py", line 65, in run     train_dataset = TextAudioSpeakerLoader(hps.data.training_files, hps)   File "Z:\FreeVCtrain24khz\data_utils_24.py", line 34, in __init__     self._filter()   File "Z:\FreeVCtrain24khz\data_utils_24.py", line 46, in _filter     lengths.append(os.path.getsize(audiopath[0]) // (2 * self.hop_length))   File "C:\Users\steven\AppData\Local\Programs\Python\Python37\lib\genericpath.py", line 50, in getsize     return os.stat(filename).st_size FileNotFoundError: [WinError 3] The system cannot find the path specified: 'DUMMY\\p337\\p337_014.wav'
I noticed this line is missing in the 24khz datautils
audiopath = audiopath[0].replace("\\","/").replace("DUMMY", "dataset/vctk-16k")

yeah it's a way to solve this error.
data_utils_24.py is a dirty code i wrote casually. the lengths, which is used to bucketize the dataset, is obtained from DUMMY dir. if the DUMMY dir exists this error won't occur. ln -s dataset/vctk-16k DUMMY or ln -s dataset/vctk-24k DUMMY or etc. can solve this too.

If I create the dummy folder, I  now get this error
 File "Z:\FreeVCtrain24khz\data_utils_24.py", line 175, in __call__     wav_padded = commons.slice_segments(wav_padded, ids_slice * 480, wav_seglen)   File "Z:\FreeVCtrain24khz\commons.py", line 53, in slice_segments     ret[i] = x[i, :, idx_str:idx_end] RuntimeError: The expanded size of the tensor (61920) must match the existing size (26880) at non-singleton dimension 1.  Target sizes: [1, 61920].  Tensor sizes: [26880]

ah sorry i forgot that spectrograms are in 16k dir. so only ln -s dataset/vctk-16k DUMMY works.

I get that error regardless of what folder I use.

I'm getting this error too. I ran all 16khz pre-processing steps then the 24khz downsample. Using the config file from the tips-for-synthesizing-24KHz-wavs-from-16kHz-wavs folder. Help pls!
./logs/freevc-24/G_0.pth
INFO:freevc-24:Loaded checkpoint './logs/freevc-24/G_0.pth' (iteration 3461)
./logs/freevc-24/D_0.pth
INFO:freevc-24:Loaded checkpoint './logs/freevc-24/D_0.pth' (iteration 3461)
INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
Traceback (most recent call last):
  File "train_24.py", line 292, in <module>
    main()
  File "train_24.py", line 49, in main
    mp.spawn(run, nprocs=n_gpus, args=(n_gpus, hps,))
  File "/home/manjaro/.conda/envs/free-vc/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 240, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/home/manjaro/.conda/envs/free-vc/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 198, in start_processes
    while not context.join():
  File "/home/manjaro/.conda/envs/free-vc/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 160, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/home/manjaro/.conda/envs/free-vc/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
    fn(i, *args)
  File "/media/manjaro/NVME_2tb/NeuralNetworks/FreeVC/train_24.py", line 115, in run
    train_and_evaluate(rank, epoch, hps, [net_g, net_d], [optim_g, optim_d], [scheduler_g, scheduler_d], scaler, [train_loader, eval_loader], logger, [writer, writer_eval])
  File "/media/manjaro/NVME_2tb/NeuralNetworks/FreeVC/train_24.py", line 136, in train_and_evaluate
    for batch_idx, items in enumerate(train_loader):
  File "/home/manjaro/.conda/envs/free-vc/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 628, in __next__
    data = self._next_data()
  File "/home/manjaro/.conda/envs/free-vc/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1333, in _next_data
    return self._process_data(data)
  File "/home/manjaro/.conda/envs/free-vc/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1359, in _process_data
    data.reraise()
  File "/home/manjaro/.conda/envs/free-vc/lib/python3.7/site-packages/torch/_utils.py", line 543, in reraise
    raise exception
RuntimeError: Caught RuntimeError in DataLoader worker process 1.
Original Traceback (most recent call last):
  File "/home/manjaro/.conda/envs/free-vc/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/manjaro/.conda/envs/free-vc/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 61, in fetch
    return self.collate_fn(data)
  File "/media/manjaro/NVME_2tb/NeuralNetworks/FreeVC/data_utils_24.py", line 174, in __call__
    wav_padded = commons.slice_segments(wav_padded, ids_slice * 480, wav_seglen)
  File "/media/manjaro/NVME_2tb/NeuralNetworks/FreeVC/commons.py", line 53, in slice_segments
    ret[i] = x[i, :, idx_str:idx_end]
RuntimeError: The expanded size of the tensor (15840) must match the existing size (8125) at non-singleton dimension 1.  Target sizes: [1, 15840].  Tensor sizes: [8125]

hmmm i think it because your 'DUMMY' ('dataset/vctk-16k') dir does not contain those 16k '.spec.pt' files.
if os.path.exists(spec_filename):
    spec = torch.load(spec_filename)
else:
    audio, sampling_rate = load_wav_to_torch(filename)
    if sampling_rate != 16000:
        raise ValueError("{} SR doesn't match target {} SR".format(
            sampling_rate, self.sampling_rate))
        audio_16 = audio / self.max_wav_value
        audio_16 = audio_16.unsqueeze(0)
    spec = spectrogram_torch(audio_16, self.filter_length,
        self.sampling_rate, self.hop_length, self.win_length,
        center=False)
    spec = torch.squeeze(spec, 0)
    torch.save(spec, spec_filename)

But the spec.pt files are present in the 16khz folder. They also appeared in the 24khz folder when I had linked that one to the dummy.
the code change you made here removes the line that points to the 24khz folder audio, sampling_rate = load_wav_to_torch(filename.replace("DUMMY", "dataset/vctk-24k"))
and if dummy is linked to 16khz, its not getting any of the 24khz files?

Still getting the error with those changes
File "Z:\FreeVCtrain24khz\data_utils_24.py", line 174, in __call__     wav_padded = commons.slice_segments(wav_padded, ids_slice * 480, wav_seglen)   File "Z:\FreeVCtrain24khz\commons.py", line 53, in slice_segments     ret[i] = x[i, :, idx_str:idx_end] RuntimeError: The expanded size of the tensor (61920) must match the existing size (33824) at non-singleton dimension 1.  Target sizes: [1, 61920].  Tensor sizes: [33824]

Still getting the error with those changes
File "Z:\FreeVCtrain24khz\data_utils_24.py", line 174, in __call__ wav_padded = commons.slice_segments(wav_padded, ids_slice * 480, wav_seglen) File "Z:\FreeVCtrain24khz\commons.py", line 53, in slice_segments ret[i] = x[i, :, idx_str:idx_end] RuntimeError: The expanded size of the tensor (61920) must match the existing size (33824) at non-singleton dimension 1. Target sizes: [1, 61920]. Tensor sizes: [33824]

Have u tried maybe removing all spec.pt files before running it again. i havent tried the changes myself yet

Ye this doesn't help
INFO:freevc-24:Loaded checkpoint './logs/freevc-24/G_0.pth' (iteration 3461)
./logs/freevc-24/D_0.pth
INFO:freevc-24:Loaded checkpoint './logs/freevc-24/D_0.pth' (iteration 3461)
INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
Traceback (most recent call last):
  File "train_24.py", line 292, in <module>
    main()
  File "train_24.py", line 49, in main
    mp.spawn(run, nprocs=n_gpus, args=(n_gpus, hps,))
  File "/home/manjaro/.conda/envs/free-vc/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 240, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/home/manjaro/.conda/envs/free-vc/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 198, in start_processes
    while not context.join():
  File "/home/manjaro/.conda/envs/free-vc/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 160, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/home/manjaro/.conda/envs/free-vc/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
    fn(i, *args)
  File "/media/manjaro/NVME_2tb/NeuralNetworks/FreeVC/train_24.py", line 115, in run
    train_and_evaluate(rank, epoch, hps, [net_g, net_d], [optim_g, optim_d], [scheduler_g, scheduler_d], scaler, [train_loader, eval_loader], logger, [writer, writer_eval])
  File "/media/manjaro/NVME_2tb/NeuralNetworks/FreeVC/train_24.py", line 136, in train_and_evaluate
    for batch_idx, items in enumerate(train_loader):
  File "/home/manjaro/.conda/envs/free-vc/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 628, in __next__
    data = self._next_data()
  File "/home/manjaro/.conda/envs/free-vc/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1333, in _next_data
    return self._process_data(data)
  File "/home/manjaro/.conda/envs/free-vc/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1359, in _process_data
    data.reraise()
  File "/home/manjaro/.conda/envs/free-vc/lib/python3.7/site-packages/torch/_utils.py", line 543, in reraise
    raise exception
RuntimeError: Caught RuntimeError in DataLoader worker process 7.
Original Traceback (most recent call last):
  File "/home/manjaro/.conda/envs/free-vc/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/manjaro/.conda/envs/free-vc/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 61, in fetch
    return self.collate_fn(data)
  File "/media/manjaro/NVME_2tb/NeuralNetworks/FreeVC/data_utils_24.py", line 180, in __call__
    wav_padded = commons.slice_segments(wav_padded, ids_slice * 480, wav_seglen)
  File "/media/manjaro/NVME_2tb/NeuralNetworks/FreeVC/commons.py", line 53, in slice_segments
    ret[i] = x[i, :, idx_str:idx_end]
RuntimeError: The expanded size of the tensor (16800) must match the existing size (13732) at non-singleton dimension 1.  Target sizes: [1, 16800].  Tensor sizes: [13732]

If I set the hop_length to 480 manually:
spec = spectrogram_torch(audio_norm, self.filter_length,
                self.sampling_rate, 480, self.win_length,
                center=False)

It seems that the sizes almost match:
RuntimeError: The expanded size of the tensor (297) must match the existing size (298) at non-singleton dimension 1.  Target sizes: [1024, 297].  Tensor sizes: [1024, 298]
self.hop_length seems to be 320 by default. Also, the audio_norm is actually the 24kHZ version of the audio. Is that supposed to be like this? I've tried importing the 16KHz versions and doing the spectrograms for them instead but the error persists.
Sorry, I'm not really smart enough for all this especially this 16 to 24KHz hack thing.

Alright, I got it to work!
I edited the data_utils_24.py like I said in post above.
Then I was thinking about it and I decided to compare the 24kHZ audio and the 16kHZ audio tracks. The 24kHZ track ended up considerably shorter than the 16 one. I'm not sure if it's because of the changes I made to the downsample_24k.py before or not but I changed it to be this and now it works and the training starts:
def process(wav_name):
    # speaker 's5', 'p280', 'p315' are excluded,
    speaker = wav_name[:4]
    wav_path = os.path.join(args.in_dir, speaker, wav_name)
    if os.path.exists(wav_path) and '.wav' in wav_path:
        os.makedirs(os.path.join(args.out_dir1, speaker), exist_ok=True)
        wav, sr = librosa.load(wav_path)
        wav, index = librosa.effects.trim(wav, top_db=20)
        peak = np.abs(wav).max()
        if peak <= 0.01:
            print(wav_name + " - Empty file?")
            return
        if peak > 1.0:
            wav = 0.98 * wav / peak
        wav = librosa.resample(wav, orig_sr=sr, target_sr=args.sr1)
        # wav1, sr = librosa.load(wav_path, sr=args.sr1)

        # wav = wav[int(index[0]*args.sr1/22050): int(index[1]*args.sr1/22050)]
        # save_name = wav_name.replace("_mic2.flac", ".wav")
        save_path1 = os.path.join(args.out_dir1, speaker, wav_name)
        wavfile.write(
            save_path1,
            args.sr1,
            (wav * np.iinfo(np.int16).max).astype(np.int16)
        )

See I noticed the short lengths right away and had already changed my downsample.py to more or less match yours.
all my file lengths match but i still get that same error.

If I set the hop_length to 480 manually:
spec = spectrogram_torch(audio_norm, self.filter_length,
                self.sampling_rate, 480, self.win_length,
                center=False)

It seems that the sizes almost match: RuntimeError: The expanded size of the tensor (297) must match the existing size (298) at non-singleton dimension 1. Target sizes: [1024, 297]. Tensor sizes: [1024, 298]
self.hop_length seems to be 320 by default. Also, the audio_norm is actually the 24kHZ version of the audio. Is that supposed to be like this? I've tried importing the 16KHz versions and doing the spectrograms for them instead but the error persists. Sorry, I'm not really smart enough for all this especially this 16 to 24KHz hack thing.

It works! Thanks a lot

@Likkkez  Can you post your full datautils_24 I changed the hop to 480 manually, but im still getting the same error.

@Likkkez Can you post your full datautils_24 I changed the hop to 480 manually, but im still getting the same error.

import time
import os
import random
import numpy as np
import torch
import torch.utils.data

import commons
from mel_processing import spectrogram_torch, spec_to_mel_torch
from utils import load_wav_to_torch, load_filepaths_and_text, transform
#import h5py

"""Multi speaker version"""
class TextAudioSpeakerLoader(torch.utils.data.Dataset):
    """
        1) loads audio, speaker_id, text pairs
        2) normalizes text and converts them to sequences of integers
        3) computes spectrograms from audio files.
    """
    def __init__(self, audiopaths, hparams):
        self.audiopaths = load_filepaths_and_text(audiopaths)
        self.max_wav_value = hparams.data.max_wav_value
        self.sampling_rate = hparams.data.sampling_rate
        self.filter_length  = hparams.data.filter_length
        self.hop_length     = hparams.data.hop_length
        self.win_length     = hparams.data.win_length
        self.use_sr = hparams.train.use_sr
        self.use_spk = hparams.model.use_spk
        self.spec_len = hparams.train.max_speclen

        random.seed(1234)
        random.shuffle(self.audiopaths)
        self._filter()

    def _filter(self):
        """
        Filter text & store spec lengths
        """
        # Store spectrogram lengths for Bucketing
        # wav_length ~= file_size / (wav_channels * Bytes per dim) = file_size / (1 * 2)
        # spec_length = wav_length // hop_length

        lengths = []
        for audiopath in self.audiopaths:
            lengths.append(os.path.getsize(audiopath[0]) // (2 * self.hop_length))
        self.lengths = lengths

    def get_audio(self, filename):
        audio, sampling_rate = load_wav_to_torch(filename.replace("DUMMY", "dataset/vctk-24k"))
        if sampling_rate != 24000:
            raise ValueError("{} SR doesn't match target {} SR".format(
                sampling_rate, self.sampling_rate))
        audio_norm = audio / self.max_wav_value
        audio_norm = audio_norm.unsqueeze(0)
        spec_filename = filename.replace(".wav", ".spec.pt")
        if os.path.exists(spec_filename):
            spec = torch.load(spec_filename)
        else:
            spec = spectrogram_torch(audio_norm, self.filter_length,
                self.sampling_rate, 480, self.win_length,
                center=False)
            spec = torch.squeeze(spec, 0)
            torch.save(spec, spec_filename)

        if self.use_spk:
            spk_filename = filename.replace(".wav", ".npy")
            spk_filename = spk_filename.replace("DUMMY", "dataset/spk")
            spk = torch.from_numpy(np.load(spk_filename))

        if not self.use_sr:
            c_filename = filename.replace(".wav", ".pt")
            c_filename = c_filename.replace("DUMMY", "dataset/wavlm")
            c = torch.load(c_filename).squeeze(0)
        else:
            i = random.randint(68,92)
            '''
            basename = os.path.basename(filename)[:-4]
            spkname = basename[:4]
            #print(basename, spkname)
            with h5py.File(f"dataset/rs/wavlm/{spkname}/{i}.hdf5","r") as f:
                c = torch.from_numpy(f[basename][()]).squeeze(0)
            #print(c)
            '''
            c_filename = filename.replace(".wav", f"_{i}.pt")
            c_filename = c_filename.replace("DUMMY", "dataset/sr/wavlm")
            c = torch.load(c_filename).squeeze(0)

        '''
        lmin = min(c.size(-1), spec.size(-1))
        spec, c = spec[:, :lmin], c[:, :lmin]
        audio_norm = audio_norm[:, :lmin*480]
        _spec, _c, _audio_norm = spec, c, audio_norm
        while spec.size(-1) < self.spec_len:
            spec = torch.cat((spec, _spec), -1)
            c = torch.cat((c, _c), -1)
            audio_norm = torch.cat((audio_norm, _audio_norm), -1)
        start = random.randint(0, spec.size(-1) - self.spec_len)
        end = start + self.spec_len
        spec = spec[:, start:end]
        c = c[:, start:end]
        audio_norm = audio_norm[:, start*480:end*480]
        '''

        if self.use_spk:
            return c, spec, audio_norm, spk
        else:
            return c, spec, audio_norm

    def __getitem__(self, index):
        return self.get_audio(self.audiopaths[index][0])

    def __len__(self):
        return len(self.audiopaths)

class TextAudioSpeakerCollate():
    """ Zero-pads model inputs and targets
    """
    def __init__(self, hps):
        self.hps = hps
        self.use_sr = hps.train.use_sr
        self.use_spk = hps.model.use_spk

    def __call__(self, batch):
        """Collate's training batch from normalized text, audio and speaker identities
        PARAMS
        ------
        batch: [text_normalized, spec_normalized, wav_normalized, sid]
        """
        # Right zero-pad all one-hot text sequences to max input length
        _, ids_sorted_decreasing = torch.sort(
            torch.LongTensor([x[0].size(1) for x in batch]),
            dim=0, descending=True)

        max_spec_len = max([x[1].size(1) for x in batch])
        max_wav_len = max([x[2].size(1) for x in batch])

        spec_lengths = torch.LongTensor(len(batch))
        wav_lengths = torch.LongTensor(len(batch))
        if self.use_spk:
            spks = torch.FloatTensor(len(batch), batch[0][3].size(0))
        else:
            spks = None

        c_padded = torch.FloatTensor(len(batch), batch[0][0].size(0), max_spec_len)
        spec_padded = torch.FloatTensor(len(batch), batch[0][1].size(0), max_spec_len)
        wav_padded = torch.FloatTensor(len(batch), 1, max_wav_len)
        c_padded.zero_()
        spec_padded.zero_()
        wav_padded.zero_()

        for i in range(len(ids_sorted_decreasing)):
            row = batch[ids_sorted_decreasing[i]]

            c = row[0]
            c_padded[i, :, :c.size(1)] = c

            spec = row[1]
            spec_padded[i, :, :spec.size(1)] = spec
            spec_lengths[i] = spec.size(1)

            wav = row[2]
            wav_padded[i, :, :wav.size(1)] = wav
            wav_lengths[i] = wav.size(1)

            if self.use_spk:
                spks[i] = row[3]

        spec_seglen = spec_lengths[-1] if spec_lengths[-1] < self.hps.train.max_speclen + 1 else self.hps.train.max_speclen + 1
        wav_seglen = spec_seglen * 480

        spec_padded, ids_slice = commons.rand_spec_segments(spec_padded, spec_lengths, spec_seglen)
        wav_padded = commons.slice_segments(wav_padded, ids_slice * 480, wav_seglen)

        c_padded = commons.slice_segments(c_padded, ids_slice, spec_seglen)[:,:,:-1]

        spec_padded = spec_padded[:,:,:-1]
        wav_padded = wav_padded[:,:,:-480]

        if self.use_spk:
          return c_padded, spec_padded, wav_padded, spks
        else:
          return c_padded, spec_padded, wav_padded

class DistributedBucketSampler(torch.utils.data.distributed.DistributedSampler):
    """
    Maintain similar input lengths in a batch.
    Length groups are specified by boundaries.
    Ex) boundaries = [b1, b2, b3] -> any batch is included either {x | b1 < length(x) <=b2} or {x | b2 < length(x) <= b3}.

    It removes samples which are not included in the boundaries.
    Ex) boundaries = [b1, b2, b3] -> any x s.t. length(x) <= b1 or length(x) > b3 are discarded.
    """
    def __init__(self, dataset, batch_size, boundaries, num_replicas=None, rank=None, shuffle=True):
        super().__init__(dataset, num_replicas=num_replicas, rank=rank, shuffle=shuffle)
        self.lengths = dataset.lengths
        self.batch_size = batch_size
        self.boundaries = boundaries

        self.buckets, self.num_samples_per_bucket = self._create_buckets()
        self.total_size = sum(self.num_samples_per_bucket)
        self.num_samples = self.total_size // self.num_replicas

    def _create_buckets(self):
        buckets = [[] for _ in range(len(self.boundaries) - 1)]
        for i in range(len(self.lengths)):
            length = self.lengths[i]
            idx_bucket = self._bisect(length)
            if idx_bucket != -1:
                buckets[idx_bucket].append(i)

        for i in range(len(buckets) - 1, 0, -1):
            if len(buckets[i]) == 0:
                buckets.pop(i)
                self.boundaries.pop(i+1)

        num_samples_per_bucket = []
        for i in range(len(buckets)):
            len_bucket = len(buckets[i])
            total_batch_size = self.num_replicas * self.batch_size
            rem = (total_batch_size - (len_bucket % total_batch_size)) % total_batch_size
            num_samples_per_bucket.append(len_bucket + rem)
        return buckets, num_samples_per_bucket

    def __iter__(self):
      # deterministically shuffle based on epoch
      g = torch.Generator()
      g.manual_seed(self.epoch)

      indices = []
      if self.shuffle:
          for bucket in self.buckets:
              indices.append(torch.randperm(len(bucket), generator=g).tolist())
      else:
          for bucket in self.buckets:
              indices.append(list(range(len(bucket))))

      batches = []
      for i in range(len(self.buckets)):
          bucket = self.buckets[i]
          len_bucket = len(bucket)
          ids_bucket = indices[i]
          num_samples_bucket = self.num_samples_per_bucket[i]

          # add extra samples to make it evenly divisible
          rem = num_samples_bucket - len_bucket
          ids_bucket = ids_bucket + ids_bucket * (rem // len_bucket) + ids_bucket[:(rem % len_bucket)]

          # subsample
          ids_bucket = ids_bucket[self.rank::self.num_replicas]

          # batching
          for j in range(len(ids_bucket) // self.batch_size):
              batch = [bucket[idx] for idx in ids_bucket[j*self.batch_size:(j+1)*self.batch_size]]
              batches.append(batch)

      if self.shuffle:
          batch_ids = torch.randperm(len(batches), generator=g).tolist()
          batches = [batches[i] for i in batch_ids]
      self.batches = batches

      assert len(self.batches) * self.batch_size == self.num_samples
      return iter(self.batches)

    def _bisect(self, x, lo=0, hi=None):
      if hi is None:
          hi = len(self.boundaries) - 1

      if hi > lo:
          mid = (hi + lo) // 2
          if self.boundaries[mid] < x and x <= self.boundaries[mid+1]:
              return mid
          elif x <= self.boundaries[mid]:
              return self._bisect(x, lo, mid)
          else:
              return self._bisect(x, mid + 1, hi)
      else:
          return -1

    def __len__(self):
        return self.num_samples // self.batch_size

Make sure that before you run this you remove all spec.pt files from ur dataset
find . -name "*.spec.pt" -type f -delete

Well it runs and starts, but after 300 steps it errors out with this
RuntimeError: The expanded size of the tensor (298) must match the existing size (299) at non-singleton dimension 1.  Target sizes: [1024, 298].  Tensor sizes: [1024, 299]

Well it runs and starts, but after 300 steps it errors out with this RuntimeError: The expanded size of the tensor (298) must match the existing size (299) at non-singleton dimension 1. Target sizes: [1024, 298]. Tensor sizes: [1024, 299]

I think that means one of your 24khz files doesnt match the 16khz version. Can you show me your downsampling scripts?

Sure
import os
import argparse
import librosa
import numpy as np
from multiprocessing import Pool, cpu_count
from scipy.io import wavfile
from tqdm import tqdm

def process(wav_name):
    # speaker 's5', 'p280', 'p315' are excluded,
    speaker = wav_name[:4]
    wav_path = os.path.join("VCTK", speaker, wav_name)
    if os.path.exists(wav_path) and ('.wav' in wav_path or '_mic2.flac' in wav_path):
        os.makedirs(os.path.join("./dataset/vctk-24k", speaker), exist_ok=True)
        wav, sr = librosa.load(wav_path,sr=48000)
        wav, index = librosa.effects.trim(wav, top_db=20)
        peak = np.abs(wav).max()
        if peak > 1.0:
            wav = 0.98 * wav / peak
        wav1 = librosa.resample(wav, orig_sr=sr, target_sr=24000)
        save_name = wav_name.replace("_mic2.flac", ".wav")
        save_path1 = os.path.join("./dataset/vctk-24k", speaker, save_name)
        wavfile.write(
            save_path1,
            24000,
            (wav1 * np.iinfo(np.int16).max).astype(np.int16)
        )

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--sr1", type=int, default=24000, help="sampling rate")
    parser.add_argument("--in_dir", type=str, default="./VCTK", help="path to source dir")
    parser.add_argument("--out_dir1", type=str, default="./dataset/vctk-24k", help="path to target dir")
    args = parser.parse_args()

    pool = Pool(processes=cpu_count()-2)

    for speaker in os.listdir("VCTK"):
        spk_dir = os.path.join("VCTK", speaker)
        if os.path.isdir(spk_dir):
            for _ in tqdm(pool.imap_unordered(process, os.listdir(spk_dir))):
                pass

The 16/22 is basically the same
def process(wav_name):
    # speaker 's5', 'p280', 'p315' are excluded,
    speaker = wav_name[:4]
    wav_path = os.path.join("VCTK", speaker, wav_name)
    if os.path.exists(wav_path) and ('.wav' in wav_path or '_mic2.flac' in wav_path):
        os.makedirs(os.path.join("./dataset/vctk-16k", speaker), exist_ok=True)
        os.makedirs(os.path.join("./dataset/vctk-22k", speaker), exist_ok=True)
        wav, sr = librosa.load(wav_path)
        wav, _ = librosa.effects.trim(wav, top_db=20)
        peak = np.abs(wav).max()
        if peak > 1.0:
            wav = 0.98 * wav / peak
        wav1 = librosa.resample(wav, orig_sr=sr, target_sr=16000)
        wav2 = librosa.resample(wav, orig_sr=sr, target_sr=22050)
        save_name = wav_name.replace("_mic2.flac", ".wav")
        save_path1 = os.path.join("./dataset/vctk-16k", speaker, save_name)
        save_path2 = os.path.join("./dataset/vctk-22k", speaker, save_name)
        wavfile.write(
            save_path1,
            16000,
            (wav1 * np.iinfo(np.int16).max).astype(np.int16)
        )
        wavfile.write(
            save_path2,
            22050,
            (wav2 * np.iinfo(np.int16).max).astype(np.int16)
        )

The 16/22 is basically the same
def process(wav_name):
    # speaker 's5', 'p280', 'p315' are excluded,
    speaker = wav_name[:4]
    wav_path = os.path.join("VCTK", speaker, wav_name)
    if os.path.exists(wav_path) and ('.wav' in wav_path or '_mic2.flac' in wav_path):
        os.makedirs(os.path.join("./dataset/vctk-16k", speaker), exist_ok=True)
        os.makedirs(os.path.join("./dataset/vctk-22k", speaker), exist_ok=True)
        wav, sr = librosa.load(wav_path)
        wav, _ = librosa.effects.trim(wav, top_db=20)
        peak = np.abs(wav).max()
        if peak > 1.0:
            wav = 0.98 * wav / peak
        wav1 = librosa.resample(wav, orig_sr=sr, target_sr=16000)
        wav2 = librosa.resample(wav, orig_sr=sr, target_sr=22050)
        save_name = wav_name.replace("_mic2.flac", ".wav")
        save_path1 = os.path.join("./dataset/vctk-16k", speaker, save_name)
        save_path2 = os.path.join("./dataset/vctk-22k", speaker, save_name)
        wavfile.write(
            save_path1,
            16000,
            (wav1 * np.iinfo(np.int16).max).astype(np.int16)
        )
        wavfile.write(
            save_path2,
            22050,
            (wav2 * np.iinfo(np.int16).max).astype(np.int16)
        )

Huh, that is quite odd. I don't see anything off about this. Maybe in the second script just change it from 22 to 24 since 22 isn't used anyway I think. So that it saves both in one script.

The 16/22 is basically the same
def process(wav_name):
    # speaker 's5', 'p280', 'p315' are excluded,
    speaker = wav_name[:4]
    wav_path = os.path.join("VCTK", speaker, wav_name)
    if os.path.exists(wav_path) and ('.wav' in wav_path or '_mic2.flac' in wav_path):
        os.makedirs(os.path.join("./dataset/vctk-16k", speaker), exist_ok=True)
        os.makedirs(os.path.join("./dataset/vctk-22k", speaker), exist_ok=True)
        wav, sr = librosa.load(wav_path)
        wav, _ = librosa.effects.trim(wav, top_db=20)
        peak = np.abs(wav).max()
        if peak > 1.0:
            wav = 0.98 * wav / peak
        wav1 = librosa.resample(wav, orig_sr=sr, target_sr=16000)
        wav2 = librosa.resample(wav, orig_sr=sr, target_sr=22050)
        save_name = wav_name.replace("_mic2.flac", ".wav")
        save_path1 = os.path.join("./dataset/vctk-16k", speaker, save_name)
        save_path2 = os.path.join("./dataset/vctk-22k", speaker, save_name)
        wavfile.write(
            save_path1,
            16000,
            (wav1 * np.iinfo(np.int16).max).astype(np.int16)
        )
        wavfile.write(
            save_path2,
            22050,
            (wav2 * np.iinfo(np.int16).max).astype(np.int16)
        )

Huh, that is quite odd. I don't see anything off about this. Maybe in the second script just change it from 22 to 24 since 22 isn't used anyway I think. So that it saves both in one script.

nvm preprocess_sr uses 22 khz

This code here gives a true 24khz file, and the top db of 30 seems to retain the plosives in the speech. With the handful of files I tested anyway.... Might be some files where 30 is still insufficient, but at 35 its not removing anything so..... As good as its gonna get I guess
def process(wav_name):
    # speaker 's5', 'p280', 'p315' are excluded,
    speaker = wav_name[:4]
    wav_path = os.path.join("VCTK", speaker, wav_name)
    if os.path.exists(wav_path) and ('.wav' in wav_path or '_mic2.flac' in wav_path):
        os.makedirs(os.path.join("./dataset/vctk-24k", speaker), exist_ok=True)
        wav, sr = librosa.load(wav_path,sr=None)
        wav, index = librosa.effects.trim(wav, top_db=30)
        peak = np.abs(wav).max()
        if peak > 1.0:
            wav = 0.98 * wav / peak
        wav1 = librosa.resample(wav, orig_sr=sr, target_sr=24000)
        save_name = wav_name.replace("_mic2.flac", ".wav")
        save_path1 = os.path.join("./dataset/vctk-24k", speaker, save_name)
        wavfile.write(
            save_path1,
            24000,
            (wav1 * np.iinfo(np.int16).max).astype(np.int16)
        )

thank you for your effort and experiments!
for the downsampling issue i think i've fixed it: link
https://github.com/OlaWod/FreeVC/blob/8e7855d89c8174735458d6160578ce303360a06b/tips-for-synthesizing-24KHz-wavs-from-16kHz-wavs/downsample_24k.py#L22-L23
a lazy way to get 24k wav with original 22k trimed index ↑
and i agree that top_db=20 is too aggressive, which removes some sounds and so makes some pronunciations bad. thank you for pointing out this!

The fix you suggested there does not work. It results in overflow issues
 RuntimeWarning: overflow encountered in long_scalars
  wav1 = wav1[int(index[0]*24000/22050): int(index[1]*24000/22050)]

All of the outputs are only 1-2 seconds long and much shorter then the 16k files.
Best solution is to downsample all files at the same time using the code I posted above but with all sample rates. This results in all necessary files being created, and all of the file lengths matching.
the problem is this here in dautautils
lengths.append(os.path.getsize(audiopath) // (2 * self.hop_length))
It is inferring the audio length from the filesize. If the header length of the wav changes then this will break, Workaround is to make sure all wavs are written by the same process that ensures a consistent header length.
so running this here is the best solution at the moment
 def process(wav_name):
    # speaker 's5', 'p280', 'p315' are excluded,
    speaker = wav_name[:4]
    wav_path = os.path.join("VCTK", speaker, wav_name)
    if os.path.exists(wav_path) and ('.wav' in wav_path or '_mic2.flac' in wav_path):
        os.makedirs(os.path.join("./dataset/vctk-16k", speaker), exist_ok=True)
        os.makedirs(os.path.join("./dataset/vctk-22k", speaker), exist_ok=True)
        os.makedirs(os.path.join("./dataset/vctk-48k", speaker), exist_ok=True)
        wav, sr = librosa.load(wav_path,sr=None)
        wav, _ = librosa.effects.trim(wav, top_db=30)
        peak = np.abs(wav).max()
        if peak > 1.0:
            wav = 0.98 * wav / peak
        wav1 = librosa.resample(wav, orig_sr=sr, target_sr=16000)
        wav2 = librosa.resample(wav, orig_sr=sr, target_sr=22050)
        wav3 = librosa.resample(wav, orig_sr=sr, target_sr=24000)
        save_name = wav_name.replace("_mic2.flac", ".wav")
        save_path1 = os.path.join("./dataset/vctk-16k", speaker, save_name)
        save_path2 = os.path.join("./dataset/vctk-22k", speaker, save_name)
        save_path3 = os.path.join("./dataset/vctk-24k", speaker, save_name)
        wavfile.write(
            save_path1,
            16000,
            (wav1 * np.iinfo(np.int16).max).astype(np.int16)
        )
        wavfile.write(
            save_path2,
            22050,
            (wav2 * np.iinfo(np.int16).max).astype(np.int16)
        )
        wavfile.write(
            save_path3,
            24000,
            (wav3 * np.iinfo(np.int16).max).astype(np.int16)
        )

yes your def process(wav_name): code is the best solution if process from scratch. i'm just lazy to reprocess all those files, i want to use my old processed files, so i get the trimed index from 22khz wav (which is what my old process did), and apply the index to 24khz wav. and this works fine for me, i did not encounter any errors, and training with these 24khz wavs is also going fine.


Change axes color, x=red, z=blue

This is more in concordance with de-facto standard
x, y, z = red, green, blue

Thanks Gaël. This actually makes sense.


Password shown in plain text on screen

If authenticating by user/password then the password should not be shown on screen.
This would be considered a security risk by governance people and would prevent it from being accepted in most places.

Thank you @pmcochrane , we're going to fix it.

Fixed on release 1.1.0


Library depends too much on the user/developer

This makes it very confusing

what is the problem? which part? what do you suggest? whats the alternative? this is not a constructive issue.

shortest full demo for browsers is 100 LOC and most of it is comments and initialization of the webcam which has nothing to do with the library
shortest full demo for nodejs is 200 LOC because of all pretty-printing of results. if you want just a quick start, here:
const fs = require('fs');
const Human = require('@vladmandic/human').default;

async function main(inputFile) {
  const human = new Human(); // create instance of human using default configuration
  const buffer = fs.readFileSync(inputFile); // read file data into buffer
  const tensor = human.tf.node.decodeImage(buffer); // decode jpg data
  const result = await human.detect(tensor); // run detection
  console.log(result);
}

main('samples/in/ai-body.jpg');


risc-v/esp32c3: Use onexit to free thread private semaphore

Summary
ESP32-C3 uses onexit to free thread private semaphore.
Impact
Fix Wi-Fi thread private semaphore can't be free when task exiting or is deleted.
Testing

@xiaoxiang781216 @gustavonihei PTAL.


Ballot Icon

Hi,
I just noticed that the ballot Icon seems broken. thought I should report it as a bug.
Thanks

working in 4.0.1
https://jossef.github.io/material-design-icons-iconfont/


Redis cache strange behavior

I'm experiencing strange behavior of distributed redis cache when combined with Thread.Sleep (or another blocking workload). Simple repro case (ASP.NET Core RC2)
Startup.cs
public void ConfigureServices(IServiceCollection services)
{
    ...

    services.AddDistributedRedisCache(options =>
    {
        options.Configuration = "...";
    });
}

HomeController.cs
public class HomeController : Controller
{
    public async Task<IActionResult> Index([FromServices]IDistributedCache cache)
    {
        var hitat = DateTime.UtcNow;
        var sw = new Stopwatch();
        sw.Start();

        var value = await cache.GetAsync("cache_key");
        sw.Stop();

        Thread.Sleep(20000);

        var rederedat = DateTime.UtcNow;
        return new ContentResult()
        {
            ContentType = "text/plain",
            Content = $"Hit at {hitat}, rendered {rederedat}, cache get duration {sw.ElapsedMilliseconds} ms"
        };
    }
}

I'm using Azure Redis Cache. When accessing this page multiple times in parallel, my results...
Hit at 7. 6. 2016 14:14:58, rendered 7. 6. 2016 14:15:18, cache get duration 510 ms
Hit at 7. 6. 2016 14:14:59, rendered 7. 6. 2016 14:15:38, cache get duration 19270 ms
Hit at 7. 6. 2016 14:15:00, rendered 7. 6. 2016 14:15:58, cache get duration 38757 ms
Hit at 7. 6. 2016 14:15:02, rendered 7. 6. 2016 14:16:18, cache get duration 56095 ms
Hit at 7. 6. 2016 14:15:04, rendered 7. 6. 2016 14:16:38, cache get duration 74689 ms

@kichalla can you repro this?

Taking a look at it now.

I am unable to repro it. I used a locally hosted redis server in my case.
@dolly22 are you able to repro it consistently?

@kichalla I am able to reproduce on another computer, even with locally hosted redis server.
Local server...
Hit at 07.06.2016 19:26:11, rendered 07.06.2016 19:26:31, cache get duration 0 ms
Hit at 07.06.2016 19:26:12, rendered 07.06.2016 19:26:51, cache get duration 18608 ms
Hit at 07.06.2016 19:26:14, rendered 07.06.2016 19:27:11, cache get duration 37174 ms
I have uploaded test project to https://github.com/dolly22/aspnetcoreredistest, also deployed to azure web app - http://aspnetcoreredistest.azurewebsites.net/.
My steps to reproduce

Open three tabs in edge browser - http://aspnetcoreredistest.azurewebsites.net/
After all tabs finish loading, hit refresh for all tabs

Thanks for the detailed repro.
We were able to isolate the issue to StackExchange's Redis client (ASP.NET's Caching uses this client behind the scenes). For example, in the following repro there is no ASP.NET Core component involved and checkout Output.txt file in the branch which shows the output that we see when running the repro.
https://github.com/kichalla/rediscacherepro/tree/stackexchangeredis/RedisRepro

I think the problem is how StackExchange's Redis client is sychronously scheduling continuations
ResultBox.cs#L110 (http://blog.stephencleary.com/2012/12/dont-block-in-asynchronous-code.html) and also requires the completion thread to finish it's work CompletionManager.cs#L124 before allowing another threads to enter.
For now, I was able to workaround the issue in my case either by setting (NOT PROPERLY TESTED)
ConnectionMultiplexer.PreserveAsyncOrder = false;

or inserting Task.Delay before synchronously blocking code
await cache.GetAsync("cache_key");
await Task.Delay(1);

OK, thanks for working that out. @dolly22 can you file a bug against stackexchange?
https://github.com/StackExchange/StackExchange.Redis/issues


The reuse problem of TCP port under multiple network connections

Hi，when I run the python code in websocketd, It shows the error “socket.error: [Errno 98] Address already in use”.
When there are multiple connections to websocketd，port 5001 is used for a welcome socket for TCP, is there a way to reuse it?
Thank you！
from sys import stdout
from socket import *
import threading
def tcplink(sock, addr):
    while 1:
        data = sock.recv(1024)
        print(data)
        stdout.flush()
    sock.close()
serverPort = 5001
serverSocket = socket(AF_INET,SOCK_STREAM)
serverSocket.bind(('',serverPort))
serverSocket.listen(5)
while 1:
    sock, addr = serverSocket.accept()
    t = threading.Thread(target=tcplink, args=(sock, addr))
    t.start()

What are you trying to do? websocketd will run multiple copies of your program. You can solve that with SO_REUSEADDR, SO_REUSEPORT but I think it might be not what you actually need.

I want to receive temperature data from a IoT device and push the temperature data to all open web pages.
Whether websocketd can run only one program to all connections, rather than multiple copies of the program，to achieve the same push for all pages.

two ways of doing it

create one script that dumps content of sensor to file every X minutes run websocket with program that dumps that file to connected clients. You might need some kind of locking.
write your own program similar to websocketd, it's relatively easy if you use it as starting point. IMO it's best choice for any serious websocket-service. websocketd is good as flexible tool that does not necessarily solves all the problem.


Terminal is unusable.

When launching code and trying to start the integrated terminal I am not able to enter text.

VSCode Version: Code 1.18.1 (929bacba01ef658b873545e26034d1a8067445e9, 2017-11-16T18:23:26.125Z)
OS Version: Darwin x64 15.6.0
Extensions:

Extension
Author (truncated)
Version

xml
Dot
1.9.2

beautify
Hoo
1.1.1

vscode-cflint
Kam
0.2.4

vscode-docker
Pet
0.0.22

gitignore
cod
0.5.0

jshint
dba
0.10.15

vscode-eslint
dba
1.4.3

gitlens
eam
6.3.0

vscode-html-css
ecm
0.1.8

vs-code-xml-format
fab
0.1.5

live-html-previewer
hdg
0.3.0

coldfusion
ili
0.0.11

mssql
ms-
1.2.1

PowerShell
ms-
1.5.1

java
red
0.14.0

jenkinsfile-support
sec
0.1.0

vscode-todo-highlight
way
0.5.11

(1 theme extensions excluded)

Steps to Reproduce:

Reproduces without extensions: Yes

So reinstalling from a download via the visual studio code site did not fix the issue but when i reinstalled via brew cask it seems to have fixed the issue.

After further testing it looks like the prompt is being pushed out of screen when the terminal window is a certain height. When I dock it to the right the prompt disappears out of screen and when I increase the height when it is docked at the bottom of the screen at about 1/3 my window size the prompt disappears out of screen.

Sorry about the delay, was on vacation. Can you post a screenshot demonstrating the issue?


Add a policy for inclusion in the prelude

Thus far, we've added items to the prelude via ad-hoc policy, and we have not
necessarily had consensus on the underlying unwritten policy, only on
the application of that unwritten policy to individual prelude proposals.
This PR adds a proposed policy for including items into the prelude. This
policy also covers when we should add items to the common prelude versus an
edition-specific prelude.

We should only add an item to the prelude if some reasonable number of crates
are likely to use the item. It need not be an item used by the majority of
crates, but it should be reasonably frequent across the ecosystem.

I think I agree with everything in this policy, but for me personally, I consider this is a necessary but sufficient criteria. In particular, in my opinion, I think there are costs to having a large prelude and I'm not sure this policy accounts for that at all. That is, as the prelude grows in size, I think our threshold for adding to it should also go up. That is, our standards should become higher.
I don't know whether there is consensus on that point such that it can be added to the policy as written though. Perhaps just the "necessary but not sufficient" language would work. I'm not sure.

@ChrisDenton wrote:

It seems that there can be some issues with name resolution when adding types to the prelude: rust-lang/rust#125107 (comment)
The crater run has Cell and Duration in the prelude. While these types probably have too a generic a name to be included, they do make for a good stress test. I think the fact that they lead to errors with disambiguation suggests we do perhaps need to exercise some caution when adding types to the prelude.

Fascinating.
I think we should leave aside the issue of conflicts caused by naming a local variable the same as a unit struct (e.g. let PhantomData = (); or let (PhantomData @ _) = ();). Those don't seem particularly likely for multiple reasons, and I'd propose that we should add them to "acceptable breakage" since there's an obvious way to avoid ever encountering that (don't name a variable LikeAType).
The issue of use EnumType::Variant; for an EnumType that conflicts with a type in the prelude is absolutely real breakage. I'm going to file an issue for about investigating that source of breakage and what we could potentially do about it: https://github.com/rust-lang/rust/issues/127738