sdmmc_read_sectors reads 512 byte sector size in 40ms - 50ms (IDFGH-7698)

Environment

Development Kit:  ESP32-Wrover-Kit

Kit version (for WroverKit/PicoKit/DevKitC): [v1|v2|v3|v4]

Module or chip used:  [ESP32-WROOM-32|

IDF version (run git describe --tags to find it):
v5.0-dev-3654-gc2ccc383da

Build System:
CMake

Compiler version (run xtensa-esp32-elf-gcc --version to find it):
xtensa-esp32-elf-gcc (crosstool-NG esp-2022r1-RC1) 11.2.0

Operating System:   Linux

Using an IDE?:  Yes, Clion C++ IDE

Power Supply:         [USB|external 5V

Problem Description
Hello, I'm trying to implement little fs on SDMMC interface. The SD card sector size is allowing 512 byte reads at a time. I think the reads function is slower because its spends 40-50 ms for 512kb data reading.
note: sd card interface's spi bus on 20Mhz. 1 Freertos task is running.
is that normal for now?, is there any way to improve that?

[18:14:28:262] Name: SD␍␊
[18:14:28:262] Type: SDHC/SDXC␍␊
[18:14:28:262] Speed: 20 MHz␍␊
[18:14:28:262] Size: 14922MB␍␊
[18:14:28:262] CSD: ver=2, sector_size=512, capacity=30560256 read_bl_len=9␍␊
[18:14:28:262] SSR: bus_width=1␍␊
[18:14:28:278] SD Card block size 512, read len 9, capacity 30560256
[READ]512 byte reading in 50 ms␍␊

Expected Behavior
Actual Behavior
Steps to reproduce

step1
...

// If possible, attach a picture of your setup/wiring here.
Code to reproduce this issue
// the code should be wrapped in the ```cpp tag so that it will be displayed better.
#include "esp_log.h"

#undef CONFIG_LITTLE_FS_IO_DEBUG
/**
 * LittleFS disk io read function
 * @param c littlefs config structure
 * @param block block address
 * @param off offset address
 * @param buffer read buffer
 * @param size read buffer size
 * @return return ok on success
 */
static int lfs_deskio_read(const struct lfs_config *c,
                           lfs_block_t block, lfs_off_t off, void *buffer, lfs_size_t size)
{
    uint32_t current_tick = xTaskGetTickCount();
#ifdef CONFIG_LITTLE_FS_IO_DEBUG
    printf("[READ]block addr %d, offset %d, size %d\n",block, off,size);
#endif
    esp_err_t ret = sdmmc_read_sectors(sdCardInstance, buffer, block, 1);
    if(ret != ESP_OK){
#ifdef CONFIG_LITTLE_FS_IO_DEBUG
        printf("[READ]Error on read, code: %d\n", ret);
#endif
        return LFS_ERR_IO;
    }
#ifdef CONFIG_LITTLE_FS_IO_DEBUG

#endif
    uint32_t elapsed = xTaskGetTickCount() - current_tick;
    elapsed = elapsed * portTICK_PERIOD_MS;
    printf("[READ]512 byte reading in %d ms\n", elapsed);
    return LFS_ERR_OK;
}

#include <string.h>
#include <sys/unistd.h>
#include <sys/stat.h>
#include <driver/rtc_io.h>
#include <esp_task_wdt.h>
#include <driver/spi_common.h>
#include <driver/sdspi_host.h>
#include <esp_log.h>

#include "sdmmc_cmd.h"
#include "lfs_port.h"
extern "C" {
void app_main(void);
}
static const char *TAG = "example";

#define MOUNT_POINT "/sdcard"

#if CONFIG_IDF_TARGET_ESP32S2
#define SPI_DMA_CHAN    host.slot
#elif CONFIG_IDF_TARGET_ESP32C3
#define SPI_DMA_CHAN    SPI_DMA_CH_AUTO
#else
#define SPI_DMA_CHAN    1
#endif

void app_main(void)
{
    rtc_gpio_hold_dis(GPIO_NUM_4);
    esp_err_t ret;
    esp_err_t err = ESP_OK;
    sdmmc_card_t sdCard;
    /*@formatter:off*/
    spi_bus_config_t spiBusConfig = {
            .mosi_io_num = GPIO_NUM_13,
            .miso_io_num = GPIO_NUM_12,
            .sclk_io_num = GPIO_NUM_14,
            .quadwp_io_num = -1,
            .quadhd_io_num = -1,
            .data4_io_num = -1,     ///< GPIO pin for spi data4 signal in octal mode, or -1 if not used.
            .data5_io_num = -1,     ///< GPIO pin for spi data5 signal in octal mode, or -1 if not used.
            .data6_io_num = -1,     ///< GPIO pin for spi data6 signal in octal mode, or -1 if not used.
            .data7_io_num = -1,     ///< GPIO pin for spi data7 signal in octal mode, or -1 if not used.
            .max_transfer_sz = 0, ///< Maximum transfer size, in bytes. Defaults to 4092 if 0 when DMA enabled, or to `SOC_SPI_MAXIMUM_BUFFER_SIZE` if DMA is disabled.
            .flags = 0,                	///< Abilities of bus to be checked by the driver. Or-ed value of ``SPICOMMON_BUSFLAG_*`` flags.
            .intr_flags = 0
    };/*@formatter:on*/

//Gonna experiment with a seperate SPI device module
    err= spi_bus_initialize(SPI2_HOST, &spiBusConfig, SPI_DMA_CH_AUTO);

    if (err != ESP_OK) {
        ESP_LOGE(TAG, "Failed to init spi2host");
    }
    sdspi_device_config_t SDSPIDeviceConfig = { .host_id = SPI2_HOST,   ///< SPI host to use, SPIx_HOST (see spi_types.h).
            .gpio_cs = static_cast<gpio_num_t>(0),     ///< GPIO number of CS signal
            .gpio_cd = GPIO_NUM_NC,     ///< GPIO number of card detect signal
            .gpio_wp = GPIO_NUM_NC,     ///< GPIO number of write protect signal
            .gpio_int = GPIO_NUM_NC,    ///< GPIO number of interrupt line (input) for SDIO card.
    };

    sdmmc_host_t SDSPIHost = SDSPI_HOST_DEFAULT();
    SDSPIHost.flags = SDMMC_HOST_FLAG_SPI;
    SDSPIHost.slot = SDSPIDeviceConfig.host_id;
    SDSPIHost.max_freq_khz = SDMMC_FREQ_DEFAULT;

    sdspi_dev_handle_t out_handle;
    sdspi_host_init();
    if( sdspi_host_init_device(&SDSPIDeviceConfig, &out_handle) == ESP_OK)
        printf("ESP host init ok\n");
    else
        printf("ESP host init fail\n");

    sdmmc_card_init(&SDSPIHost, &sdCard);
    // Card has been initialized, print its properties
#if LOG_LOCAL_LEVEL > ESP_LOG_WARN
    sdmmc_card_print_info(stdout, &sdCard);
#endif
    printf("SD Card block size %d, read len %d, capacity %d\n",
           sdCard.csd.sector_size,
           sdCard.csd.read_block_len,
           sdCard.csd.capacity);

     LittleFS_Mount(&sdCard);
    for(;;){
        vTaskDelay(100);
    }
}

Debug Logs
[17:53:28:418] [READ]512 byte reading time 50 ms␍␊
[17:53:28:466] [READ]512 byte reading time 40 ms␍␊
[17:53:28:753] [READ]512 byte reading time 40 ms␍␊
[17:53:28:785] [READ]512 byte reading time 40 ms␍␊
[17:53:28:833] [READ]512 byte reading time 40 ms␍␊
[17:53:29:025] [READ]512 byte reading time 50 ms␍␊
[17:53:29:152] [READ]512 byte reading time 40 ms␍␊
[17:53:29:360] [READ]512 byte reading time 40 ms␍␊

Changed SD card frequency to 40MHZ but no luck :(
[18:37:30:266] Name: SD␍␊
[18:37:30:266] Type: SDHC/SDXC␍␊
[18:37:30:266] Speed: 40 MHz␍␊
[18:37:30:266] Size: 14922MB␍␊
[18:37:30:266] CSD: ver=2, sector_size=512, capacity=30560256 read_bl_len=9␍␊
[18:37:30:266] SSR: bus_width=1␍␊
[18:37:30:266] SD Card block size 512, read len 9, capacity 30560256␍␊
[18:37:30:282] [READ]512 byte reading in 40 ms␍␊
[18:37:30:330] [READ]512 byte reading in 50 ms␍␊
[18:37:30:378] [READ]512 byte reading in 40 ms␍␊
[18:37:30:409] [READ]512 byte reading in 40 ms␍␊

Allocated dma buffers for sd card but no luck:(
void* tmp_buf = heap_caps_malloc(512, MALLOC_CAP_DMA);
[18:48:37:310] [READ]512 byte reading in 40 ms␍␊
[18:48:37:358] dma capable␍␊
[18:48:37:358] [READ]512 byte reading in 40 ms␍␊
[18:48:37:646] dma capable␍␊

#3249

[19:31:47:979] Name: SD␍␊
[19:31:47:979] Type: SDHC/SDXC␍␊
[19:31:47:979] Speed: 40 MHz␍␊
[19:31:47:979] Size: 14922MB␍␊
[19:31:48:022] CSD: ver=2, sector_size=512, capacity=30560256 read_bl_len=9␍␊
[19:31:48:022] SSR: bus_width=1␍␊
[19:31:48:022] SD Card block size 512, read len 9, capacity 30560256␍␊
[19:31:48:022] <0x1b>[0;32mI (1293) example: SD card info:<0x1b>[0m␍␊
[19:31:48:022] <0x1b>[0;32mI (1303) example: ⇥	Bus width (log2): 0<0x1b>[0m␍␊
[19:31:48:022] <0x1b>[0;32mI (1303) example: ⇥	Freq (kHz): 40000<0x1b>[0m␍␊
[19:31:48:022] <0x1b>[0;32mI (1303) example: ⇥	DDR: 0<0x1b>[0m␍␊
[19:31:48:022] <0x1b>[0;32mI (1313) example: ⇥	CID: Date 338, MFG_ID 18, Name SD, OEM ID 13398, Rev 0, Serial 3271<0x1b>[0m␍␊
[19:31:48:022] <0x1b>[0;32mI (1323) example: ⇥	CSD: Capacity 30560256, Card Common Class 1461, CSD version 1, MMC version 0, read block len 9, sector size 512, tr speed 50000000<0x1b>[0m␍␊
[19:31:48:070] <0x1b>[0;32mI (1333) example: ⇥	CSD: Ease mem state 0, Power class 0, Revision 0, Sec feature 0<0x1b>[0m␍␊
[19:31:48:070] <0x1b>[0;32mI (1343) example: ⇥	SCR: bus width 5, erase mem state 1, reserved 0, rsvd_mnf 0, sd_spec 2<0x1b>[0m␍␊
[19:31:48:070] <0x1b>[0;32mI (1353) example: ⇥	SSR: cur_bus_width 0, discard_support 0, fule_support 0, reserved 0<0x1b>[0m␍␊

i have ported littlefs over sd card but i have to improve that read speed.
example code for anyone.
https://github.com/muratdemirtas/sqlite3-esp32-littlefs-sdcard-sdspi

in esp-idf-4.4 everything is ok.
21:02:26:874] [SQLite3]File position set ok, pos 2227200␍␊ [21:02:26:874] [SQLite3]Current offset 2227200, required amount 512␍␊ [21:02:26:939] [SQLite3]Readed 512 bytes␍␊ [21:02:26:939] esp32_Read: 3r JanStore.db 512 512 OK␍␊ [21:02:26:939] esp32_Read: 1r JanStore.db<PHONE_NUMBER>[2227712] ␍␊ [21:02:26:939] [SQLite3]File position set ok, pos 2227712␍␊ [21:02:26:939] [SQLite3]Current offset 2227712, required amount 512␍␊ [21:02:26:939] [SQLite3]Readed 512 bytes␍␊ [21:02:26:939] esp32_Read: 3r JanStore.db 512 512 OK␍␊ [21:02:26:939] esp32_Read: 1r JanStore.db<PHONE_NUMBER>[2228224] ␍␊ [21:02:26:939] [SQLite3]File position set ok, pos 2228224␍␊ [21:02:26:939] [SQLite3]Current offset 2228224, required amount 512␍␊ [21:02:26:939] [SQLite3]Readed 512 bytes␍␊ [21:02:26:939] esp32_Read: 3r JanStore.db 512 512 OK␍␊ [21:02:26:939] esp32_Read: 1r JanStore.db<PHONE_NUMBER>[2228736] ␍␊ [21:02:26:987] [SQLite3]File position set ok, pos 2228736␍␊ [21:02:26:987] [SQLite3]Current offset 2228736, required amount 512␍␊ [21:02:26:987] [SQLite3]Readed 512 bytes␍␊ [21:02:26:987] esp32_Read: 3r JanStore.db 512 512 OK␍␊ [21:02:26:987] esp32_Read: 1r JanStore.db<PHONE_NUMBER>[2229248] ␍␊ [21:02:26:987] [SQLite3]File position set ok, pos 2229248␍␊ [21:02:26:987] [SQLite3]Current offset 2229248, required amount 512␍␊ [21:02:26:987] [SQLite3]Readed 512 bytes␍␊

it was a spi ram allocation issue. The mmc driver using dma memory for communication with sd card. If your device allocates dma from spi ram then the esp first creating a memory block in spi ram( THIS IS THE ISSUE). 2x spi communication makes communication slower.  Thanks.

it was a spi ram allocation issue. The mmc driver using dma memory for communication with sd card. If your device allocates dma from spi ram then the esp first creating a memory block in spi ram( THIS IS THE ISSUE) which 2x spi communication makes communication slower. Thanks.

So how exactly you solved this issue? I assume that you somehow disabled using memory block in spi ram?


AVRO-3402 Updated GenericWriter to meet style guidelines

Make sure you have checked all steps below.
Jira

[ ] My PR addresses the following Avro Jira issues and references them in the PR title. For example, "AVRO-1234: My Avro PR"

https://issues.apache.org/jira/browse/AVRO-3402
Parent Story: https://issues.apache.org/jira/browse/AVRO-3383

Tests

[ ] My PR does not need testing for this extremely good reason: non functional changes

Commits

[ ] My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters (not including Jira issue reference)
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Documentation

[ ] In case of new functionality, my PR adds documentation that describes how to use it.

All the public functions and the classes in the PR contain Javadoc that explain what it does

@KyleSchoonover @martin-g @RyanSkraba
I start to loose confidence in myself that this is the right way to handle this. I still beleive that this effort should be grouped by warning type and not by a massive to rework all warnings. IMO formatting warnings can be fixed easily, however changing massive code for reworking conditions and  casting seems dangerous to me. It is hard to review so much code and the compiler suggestion IMO cannot be trusted 100%.
Any comments?

E.g. some changes like this make me worried:
if (o1 is IDictionary)
vs
if (left is Array leftArray)

IMO that is not trivial to me if it is correct. Even if the compiler says ok, and the init tests pass, that is a risky change.

This is actually the recommended way to avoid additional casting.  Not only a styling thing, but a performance improvement.

I am not questening the additional casting. That makes sense. my issue is using Array instead of IDictionary

I am not questening the additional casting. That makes sense. my issue is using Array instead of IDictionary

I'm a bit confused then, are you talking about a specific line in the code?

IMO the styling should be grouped by the actual warnings. E.g.

PR(s) Fixing IDE-00055 (fix formatting)
PR(s) Fixing IDE-0001 - IDE0005
...

Those PRs can cover many file, since the changes are very similar, however if all changes are in one PR and e.g. the diff is nit trivial, the review process can take very significant time

@KyleSchoonover I commented in the code, where I saw that misleading diff with IDioctionary vs Array.

My expectation was that first the ticket for #1534 will be finalized and then the rework will be started and not the other way around


ProducerAsync not responding

Description
I have an abstraction assembly which uses the Confluent.Kafka.dll (nuget) for making our developers one point of access, but i realized  that when im publishing it, everyone must add the System.Runtime.CompilerServices.Unsafe.dll to their project, although its already referenced in my assembly, when nuget the CK.dll. If they dont, the pdocuer sends that message but not returning a result.
can you explain?
How to reproduce
Create an assembly and add the Confluent.Kafka.dll (nuget) to the project, expose a produce method which implement the CK.dll producer method and return a  result.
Create a console app that has a reference to the new assembly and try to produce message.
Should not returning a result.
Add to the assembly the System.Runtime.CompilerServices.Unsafe.dll (nuget), now should work.
Checklist
Please provide the following information:

[ 0.11.5] Confluent.Kafka nuget version:
[ ] Apache Kafka version:
[ ] Client configuration:
[ ] Operating system:
[ ] Provide logs (with "debug" : "..." as necessary in configuration)
[ ] Provide broker log excerpts
[ ] Critical issue

I have found that you don't actually need to add System.Runtime.CompilerServices.Unsafe.dll to executable assembly (main/exe project), but have assembly version binding in App.config:
<configuration>
	<runtime>
		<assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
			<dependentAssembly>
				<assemblyIdentity name="System.Runtime.CompilerServices.Unsafe" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
				<bindingRedirect oldVersion="<IP_ADDRESS>-<IP_ADDRESS>" newVersion="<IP_ADDRESS>" />
			</dependentAssembly>
		</assemblyBinding>
	</runtime>
</configuration>

There is no hope with that. my team which reference my DLL which calls your DLL, in win forms and Web API facing the issue regardless adding the unsafe binding. from my point of view this is a big issue that no one pay attention to it

I'll try to replicate before the 1.0 release.
note: if you are on windows, you should not use 0.11.5 due to known issues - upgrade to 0.11.6 or higher.

thank you for the quick response. im using 0.11.6


Crash when run as administrator for packaged app

Describe the bug
My App is a packages App, in the main code I called FindOrRegisterForKey to achieve singleton.
however, the Class not registered excpetion happened when I run the App as Administrator...
run as user works just fine.
OSVer:  Windows 10.0.18363.418
AppSDKversion: Microsoft.WindowsAppRuntime.1.1_1004.584.2120.0_x86__8wekyb3d8bbwe
StackTrace:

CoreCLR Version: 6.0.722.32202
.NET Version: 6.0.7
Description: The process was terminated due to an unhandled exception.
Exception Info: System.TypeInitializationException: The type initializer for 'WinRT.ActivationFactory1' threw an exception. ---> System.Runtime.InteropServices.COMException (0x80040154): Class not registered (0x80040154 (REGDB_E_CLASSNOTREG))    at System.Runtime.InteropServices.Marshal.ThrowExceptionForHR(Int32 errorCode)    at WinRT.BaseActivationFactory..ctor(String typeNamespace, String typeFullName)    at WinRT.ActivationFactory1..ctor()
at WinRT.ActivationFactory`1..cctor()
--- End of inner exception stack trace ---
at Microsoft.Windows.AppLifecycle.AppInstance.Make___objRef_global__Microsoft_Windows_AppLifecycle_IAppInstanceStatics()
at Microsoft.Windows.AppLifecycle.AppInstance.get__objRef_global__Microsoft_Windows_AppLifecycle_IAppInstanceStatics()
at Microsoft.Windows.AppLifecycle.AppInstance.FindOrRegisterForKey(String key)
at LenovoVantage.Program.Main(String[] args)

Steps to reproduce the bug
just launch... the app.
Expected behavior
launch normally.
Screenshots
No response
NuGet package version
1.1.4
Packaging type
Packaged (MSIX)
Windows version
Windows 10 version 1909 (18363, November 2019 Update)
IDE
Visual Studio 2022
Additional context
No response

just figured, the OS is not in elevation support scope. I just switch off to a different solution.


ES-1477 Add default metrics

Enables the following metrics by default:

corda_flow_execution_time_seconds_(count|sum|max)
corda_http_server_request_time_seconds_(count|sum|max)
corda_p2p_gateway_inbound_request_time_seconds_(count|sum|max)
corda_p2p_gateway_outbound_request_time_seconds_(count|sum|max)
corda_p2p_gateway_outbound_tls_connections_total
corda_p2p_message_outbound_total
corda_p2p_message_outbound_replayed_total
corda_p2p_message_outbound_latency_seconds_(count|sum|max)
corda_p2p_message_inbound_total
corda_p2p_session_outbound_total
corda_p2p_session_inbound_total
corda_membership_actions_handler_time_seconds_(count|sum|max)
jvm_.*
process_cpu_usage

Note that _bucket metrics are not captured as the number of buckets tends to mean these have higher cardinality.
Adds the ability to drop labels (across all metrics) and, by default, drops the following labels that can lead to high cardinality:

vitualnode_destination
virtualnode_source

Screenshot showing the reduced set of metrics post-install:


Direct imports of methods and classes prevents them from being mocked

Imports like this:
from cirq.google.decompositions import two_qubit_matrix_to_native_gates
where two_qubit_matrix_to_native_gates is a function in module cirq.google.decompositions, makes impossible to use unittest.mock.patch on it and mock this function.
While testing no error appears and the original function is called.
This kind of direct imports (for both functions and classes) should be most likely avoided and present code migrated, since this problem spreads to the users of the module importing dependencies this way.

Can we write a test to catch these?  I don't know how many we have, so trying to write a test to spot them would be a good start.

I had a feeling that at some point mocking was not an encouraged testing practice in Cirq. Is that still the case?

In core Cirq I think this is true.
But for interface boundaries it is good, so for example Engine code uses mocking.  Another place that it seems ok is for "more complex" abstract classes which want to test that "more complex" by mocking out the abstract classes.

Closing this because it's more of a local implementation detail / development guideline than a bug / feature request.


Tensorflow crashes due to abnormally high GPU RAM allocation

I have installed spleeter through poetry.
I activate the poetry virtual env where all the dependencies have been installed.
When I run the spleeter test command I get this long rant and I can see that my GPU RAM usage goes up and max out at 23.7GB (I have a 24GB GPU card, which I think should be enough):
(spleeter-iG7E_J6Q-py3.8) tetsfr@tetsfr:~/spleeter$ spleeter separate -o output audio_example.mp3
Traceback (most recent call last):
  File "/home/tetsfr/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 1375, in _do_call
    return fn(*args)
  File "/home/tetsfr/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 1359, in _run_fn
    return self._call_tf_sessionrun(options, feed_dict, fetch_list,
  File "/home/tetsfr/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 1451, in _call_tf_sessionrun
    return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
  (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[{{node conv2d/Conv2D}}]]
	 [[strided_slice_23/_907]]
  (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[{{node conv2d/Conv2D}}]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tetsfr/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/bin/spleeter", line 5, in <module>
    entrypoint()
  File "/home/tetsfr/spleeter/spleeter/__main__.py", line 256, in entrypoint
    spleeter()
  File "/home/tetsfr/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/typer/main.py", line 214, in __call__
    return get_command(self)(*args, **kwargs)
  File "/home/tetsfr/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/tetsfr/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/tetsfr/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/tetsfr/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/tetsfr/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/tetsfr/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "/home/tetsfr/spleeter/spleeter/__main__.py", line 128, in separate
    separator.separate_to_file(
  File "/home/tetsfr/spleeter/spleeter/separator.py", line 378, in separate_to_file
    sources = self.separate(waveform, audio_descriptor)
  File "/home/tetsfr/spleeter/spleeter/separator.py", line 319, in separate
    return self._separate_tensorflow(waveform, audio_descriptor)
  File "/home/tetsfr/spleeter/spleeter/separator.py", line 301, in _separate_tensorflow
    prediction = next(prediction_generator)
  File "/home/tetsfr/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 631, in predict
    preds_evaluated = mon_sess.run(predictions)
  File "/home/tetsfr/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/tensorflow/python/training/monitored_session.py", line 775, in run
    return self._sess.run(
  File "/home/tetsfr/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/tensorflow/python/training/monitored_session.py", line 1280, in run
    return self._sess.run(
  File "/home/tetsfr/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/tensorflow/python/training/monitored_session.py", line 1385, in run
    raise six.reraise(*original_exc_info)
  File "/home/tetsfr/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/six.py", line 703, in reraise
    raise value
  File "/home/tetsfr/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/tensorflow/python/training/monitored_session.py", line 1370, in run
    return self._sess.run(*args, **kwargs)
  File "/home/tetsfr/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/tensorflow/python/training/monitored_session.py", line 1438, in run
    outputs = _WrappedSession.run(
  File "/home/tetsfr/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/tensorflow/python/training/monitored_session.py", line 1201, in run
    return self._sess.run(*args, **kwargs)
  File "/home/tetsfr/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 967, in run
    result = self._run(None, fetches, feed_dict, options_ptr,
  File "/home/tetsfr/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 1190, in _run
    results = self._do_run(handle, final_targets, final_fetches,
  File "/home/tetsfr/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 1368, in _do_run
    return self._do_call(_run_fn, feeds, fetches, targets, options,
  File "/home/tetsfr/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 1394, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
  (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[node conv2d/Conv2D (defined at /spleeter/spleeter/model/functions/unet.py:109) ]]
	 [[strided_slice_23/_907]]
  (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[node conv2d/Conv2D (defined at /spleeter/spleeter/model/functions/unet.py:109) ]]
0 successful operations.
0 derived errors ignored.

Errors may have originated from an input operation.
Input Source operations connected to node conv2d/Conv2D:
 strided_slice_3 (defined at /spleeter/spleeter/model/__init__.py:305)

Input Source operations connected to node conv2d/Conv2D:
 strided_slice_3 (defined at /spleeter/spleeter/model/__init__.py:305)

Original stack trace for 'conv2d/Conv2D':
  File "/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/bin/spleeter", line 5, in <module>
    entrypoint()
  File "/spleeter/spleeter/__main__.py", line 256, in entrypoint
    spleeter()
  File "/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/typer/main.py", line 214, in __call__
    return get_command(self)(*args, **kwargs)
  File "/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "/spleeter/spleeter/__main__.py", line 128, in separate
    separator.separate_to_file(
  File "/spleeter/spleeter/separator.py", line 378, in separate_to_file
    sources = self.separate(waveform, audio_descriptor)
  File "/spleeter/spleeter/separator.py", line 319, in separate
    return self._separate_tensorflow(waveform, audio_descriptor)
  File "/spleeter/spleeter/separator.py", line 301, in _separate_tensorflow
    prediction = next(prediction_generator)
  File "/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 612, in predict
    estimator_spec = self._call_model_fn(features, None, ModeKeys.PREDICT,
  File "/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1163, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "/spleeter/spleeter/model/__init__.py", line 568, in model_fn
    return builder.build_predict_model()
  File "/spleeter/spleeter/model/__init__.py", line 516, in build_predict_model
    tf.estimator.ModeKeys.PREDICT, predictions=self.outputs
  File "/spleeter/spleeter/model/__init__.py", line 318, in outputs
    self._build_outputs()
  File "/spleeter/spleeter/model/__init__.py", line 499, in _build_outputs
    self._outputs = self._build_output_waveform(self.masked_stfts)
  File "/spleeter/spleeter/model/__init__.py", line 342, in masked_stfts
    self._build_masked_stfts()
  File "/spleeter/spleeter/model/__init__.py", line 465, in _build_masked_stfts
    for instrument, mask in self.masks.items():
  File "/spleeter/spleeter/model/__init__.py", line 336, in masks
    self._build_masks()
  File "/spleeter/spleeter/model/__init__.py", line 432, in _build_masks
    output_dict = self.model_outputs
  File "/spleeter/spleeter/model/__init__.py", line 312, in model_outputs
    self._build_model_outputs()
  File "/spleeter/spleeter/model/__init__.py", line 211, in _build_model_outputs
    self._model_outputs = apply_model(
  File "/spleeter/spleeter/model/functions/unet.py", line 197, in unet
    return apply(apply_unet, input_tensor, instruments, params)
  File "/spleeter/spleeter/model/functions/__init__.py", line 44, in apply
    output_dict[out_name] = function(
  File "/spleeter/spleeter/model/functions/unet.py", line 109, in apply_unet
    conv1 = conv2d_factory(conv_n_filters[0], (5, 5))(input_tensor)
  File "/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 783, in __call__
    outputs = call_fn(cast_inputs, *args, **kwargs)
  File "/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/tensorflow/python/keras/layers/convolutional.py", line 249, in call
    outputs = self._convolution_op(inputs, self.kernel)
  File "/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/tensorflow/python/util/dispatch.py", line 206, in wrapper
    return target(*args, **kwargs)
  File "/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/tensorflow/python/ops/nn_ops.py", line 1012, in convolution_v2
    return convolution_internal(
  File "/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/tensorflow/python/ops/nn_ops.py", line 1142, in convolution_internal
    return op(
  File "/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/tensorflow/python/ops/nn_ops.py", line 2596, in _conv2d_expanded_batch
    return gen_nn_ops.conv2d(
  File "/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 969, in conv2d
    _, _, _op, _outputs = _op_def_library._apply_op_helper(
  File "/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/tensorflow/python/framework/op_def_library.py", line 748, in _apply_op_helper
    op = g._create_op_internal(op_type_name, inputs, dtypes=None,
  File "/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 3557, in _create_op_internal
    ret = Operation(
  File "/.cache/pypoetry/virtualenvs/spleeter-iG7E_J6Q-py3.8/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 2045, in __init__
    self._traceback = tf_stack.extract_stack_for_node(self._c_op)

I have read that it may come from tf not seeing cudnn but when check my python3 environment from the virtual env, I do not see any tf issue at load time:
(spleeter-iG7E_J6Q-py3.8) tetsfr@tetsfr:~/spleeter$ python3
Python 3.8.10 (default, Sep 28 2021, 16:10:42)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
2021-11-15 02:02:06.197757: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
>>> tf.compat.v1.Session()
2021-11-15 02:03:04.095697: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
2021-11-15 02:03:04.135577: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-11-15 02:03:04.135973: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA GeForce RTX 3090 computeCapability: 8.6
coreClock: 1.785GHz coreCount: 82 deviceMemorySize: 23.69GiB deviceMemoryBandwidth: 871.81GiB/s
2021-11-15 02:03:04.135990: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2021-11-15 02:03:04.138273: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
2021-11-15 02:03:04.138298: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
2021-11-15 02:03:04.139412: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10
2021-11-15 02:03:04.139556: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10
2021-11-15 02:03:04.139846: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.11
2021-11-15 02:03:04.140356: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11
2021-11-15 02:03:04.140426: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
2021-11-15 02:03:04.140478: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-11-15 02:03:04.140889: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-11-15 02:03:04.141260: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2021-11-15 02:03:04.141474: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-11-15 02:03:04.142074: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-11-15 02:03:04.142526: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA GeForce RTX 3090 computeCapability: 8.6
coreClock: 1.785GHz coreCount: 82 deviceMemorySize: 23.69GiB deviceMemoryBandwidth: 871.81GiB/s
2021-11-15 02:03:04.142569: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-11-15 02:03:04.142973: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-11-15 02:03:04.143415: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2021-11-15 02:03:04.143435: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2021-11-15 02:03:04.443087: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-11-15 02:03:04.443111: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264]      0
2021-11-15 02:03:04.443116: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0:   N
2021-11-15 02:03:04.443216: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-11-15 02:03:04.443612: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-11-15 02:03:04.443977: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-11-15 02:03:04.444332: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15645 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:01:00.0, compute capability: 8.6)
<tensorflow.python.client.session.Session object at 0x7faa7a0e1e50>

Any idea on what is wrong?
Could that come from my CUDA v11.3 version and some issues with TF2.5 or with Spleeter?
On my regular  python3 environment, I have TF2.8 installed and it works fine with other repo I have tried, but from experience TF is a backward compatible nightmare so, hum yeah.
Thanks for your help

OK,
so after some more search I found a solution which worked like magic.
I needed to set a sys variable to true and then no more TF memory crash.
export TF_FORCE_GPU_ALLOW_GROWTH=true
(spleeter-iG7E_J6Q-py3.8) tetsfr@tetsfr:~/spleeter$ spleeter separate -o output audio_example.mp3
INFO:spleeter:File output/audio_example/vocals.wav written succesfully
INFO:spleeter:File output/audio_example/accompaniment.wav written succesfully
Oh boy, I hate so much TF. Problem solved anyway, closing this.


Ignore temporary buffers with buftype set

Resolves #100

Many thanks for the patch!  I will get that released in the next couple of days.


Log the --match argument during reprocessing as well

Previously they weren't logged for variables being locally reprocessed. Also tweaked the formatting of the logging in __main__ to be more consistent with reprocess().

LGTM


feat: add logic to create files based on plugin type

Description

After splitting the content of queries into two different files, we are storing everything as text files. But we can identify the PlugIn and use the appropriate file extension while writing to the file system.
Ex - DB type plugins have .sql files

Fixes #21371
Type of change

New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Manual

Test Plan

Add Testsmith test cases links that relate to this PR

Issues raised during DP testing

Link issues raised during DP testing for better visiblity and tracking (copy link from comments dropped on this PR)

Checklist:
Dev activity

[ ] My code follows the style guidelines of this project
[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[ ] PR is being merged under a feature flag

QA activity:

[ ] Test plan has been approved by relevant developers
[ ] Test plan has been peer reviewed by QA
[ ] Cypress test cases have been added and approved by either SDET or manual QA
[ ] Organized project review call with relevant stakeholders after Round 1/2 of QA
[ ] Added Test Plan Approved label after reveiwing all Cypress test

/build-deploy-preview

/ok-to-test

@AnaghHegde adding APIs is not coming up in git sync modal, also the progress bar doesn't show while committing these changes. Refer:
https://user-images.githubusercontent.com/80334441/225686360-08a2dc49-d0c1-46e1-a64d-10342fbe9e11.mov

This will be merged along with https://github.com/appsmithorg/appsmith/pull/21911
We made this decision to avoid the Migration changes faced by the user twice.
cc - @infinitetrooper

Closing this for the time


Windows 8 compatible

Could you make it windows 8 compatible, or is there something that prevents it from being windows 8 compatible.

I've run it on Windows 7 with no issues. From my understanding, it's just not actively being tested on any other Windows platform than 10. Have you had any issues with running it on Win8?

I found the problem. I was running python 3.4 when I needed to run python 2.7.


Timeout rejections are unhandled

In this code:
https://github.com/amplitude/experiment-js-client/blob/05c2c3898cd7dcac4b60ae4794dd73c7e7004be3/packages/experiment-browser/src/transport/http.ts#L18-L36
The timeout rejections will be unhandled, polluting the console with errors, and error reporting tools such as Sentry get flooded with useless errors.
The comment on GH from which this function was copied from states for this to be safe, but indeed it is not.
A small repro is this codesandbox.
A less naive implementation can be found in this package: https://github.com/sindresorhus/p-timeout

@moltar, many thanks for raising this issue. To reduce error-log pollution, the specific timeout errors are now logged at the debug-level in v1.12.0. Please let us know if this issue persists!

I don't think it addresses the main problem - which is unhandled exceptions due to naive handling of the promises.
Please refer to my repro example.

Hi @moltar, thanks again for submitting this issue.
Question regarding the code sandbox example you provided.
try {
  timeout(sleep(2_000), 1_000);
} catch {
  console.log("timeout");
}

Is the expected behavior that the catch statement is run?
If so, this is expected to not occur since the timeout function returns a promise. To catch the error from the promise, we either need to await the promise returned by timeout with the existing try/catch, or use a .catch() function chained to timeout(...).
The caller of request which calls timeout is set up to handle the errors from the rejected promises. Previously we were logging all these errors including the timeouts as error logs, but now we're logging them as debug.
Does this resolve the issue? Or did I miss something with respect to how the timeout function returns it's errors?

@moltar thank you for your patience regarding this issue. It is possible that the continued Sentry errors are due to timeout errors not being caught for flag fetches when the client is not in debug-logging mode. v1.12.3 will now always catch timeout errors, regardless of logging mode. Please let us know if your issue persists.


:construction: WIP : setup prometheus through Docker

this PR is related to #173
there is some stuff left to do. Among others :

setup docker installalation tasks in a dedicated install.yml file (as discussed in the issue)
setup / adapt molecule to test installation with Docker

ATM, fluke8 seems to be broke. It runs now and as for tasks in blocks to be indent with 2 more spaces, but if I do so, I got this :
--> Executing Flake8 on files found in /home/vagrant/ansible-prometheus/molecule/alternative/tests/...
Traceback (most recent call last):
  File "/home/vagrant/ansible-prometheus/.tox/py27-ansible27/bin/flake8", line 6, in <module>
    from flake8.main.cli import main
  File "/home/vagrant/ansible-prometheus/.tox/py27-ansible27/local/lib/python2.7/site-packages/flake8/main/cli.py", line 2, in <module>
    from flake8.main import application
  File "/home/vagrant/ansible-prometheus/.tox/py27-ansible27/local/lib/python2.7/site-packages/flake8/main/application.py", line 14, in <module>
    from flake8.main import options
  File "/home/vagrant/ansible-prometheus/.tox/py27-ansible27/local/lib/python2.7/site-packages/flake8/main/options.py", line 4, in <module>
    from flake8.main import vcs
  File "/home/vagrant/ansible-prometheus/.tox/py27-ansible27/local/lib/python2.7/site-packages/flake8/main/vcs.py", line 4, in <module>
    from flake8.main import mercurial
  File "/home/vagrant/ansible-prometheus/.tox/py27-ansible27/local/lib/python2.7/site-packages/flake8/main/mercurial.py", line 7, in <module>
    import configparser
  File "/home/vagrant/ansible-prometheus/.tox/py27-ansible27/local/lib/python2.7/site-packages/configparser.py", line 12, in <module>
    from backports.configparser import (
  File "/home/vagrant/ansible-prometheus/.tox/py27-ansible27/local/lib/python2.7/site-packages/backports/configparser/__init__.py", line 139, in <module>
    from backports.configparser.helpers import OrderedDict as _default_dict
  File "/home/vagrant/ansible-prometheus/.tox/py27-ansible27/local/lib/python2.7/site-packages/backports/configparser/helpers.py", line 193, in <module>
    dict(__metaclass__=abc.ABCMeta),
  File "/home/vagrant/ansible-prometheus/.tox/py27-ansible27/lib/python2.7/abc.py", line 87, in __new__
    cls = super(ABCMeta, mcls).__new__(mcls, name, bases, namespace)
TypeError: type() argument 1 must be string, not newstr
An error occurred during the test sequence action: 'lint'. Cleaning up.

I'll do some comments inside the changes I made to explain a bit more clearly what I did here and there if there is any question about it
Also, I'm always looking for feedbacks so any comments are welcomed ! :smile:

I'm trying to get on the molecule side of things to test an installation with docker.
ATM, this is not working as I'm facing an issue with molecule (probably nothing really problematic but my limited knowlege with molecule blocks me for now)
The problem appear when molecule is in the converge action and complain that he can't get facts :
    TASK [Gathering Facts] *********************************************************
    fatal: [docker]: UNREACHABLE! => {"changed": false, "msg": "Authentication or permission failure. In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in \"/tmp\". Failed command was: ( umask 77 && mkdir -p \"` echo ~/.ansible/tmp/ansible-tmp-1549558006.05-243433001157362 `\" && echo ansible-tmp-1549558006.05-243433001157362=\"` echo ~/.ansible/tmp/ansible-tmp-1549558006.05-243433001157362 `\" ), exited with result 1", "unreachable": true}

This is because the container he tries to connect to has been destroy just before. Nothing on molecule tell me that though. Here's a more full trace of execution :
vagrant@ubuntu-xenial:~/ansible-prometheus$ molecule test -s docker
--> Validating schema /home/vagrant/ansible-prometheus/molecule/default/molecule.yml.
Validation completed successfully.
--> Test matrix

└── docker
    ├── lint
    ├── destroy
    ├── dependency
    ├── syntax
    ├── create
    ├── prepare
    ├── converge
    ├── idempotence
    ├── side_effect
    ├── verify
    └── destroy

--> Scenario: 'docker'
--> Action: 'lint'
--> Executing Yamllint on files found in /home/vagrant/ansible-prometheus/...
Lint completed successfully.
--> Executing Flake8 on files found in /home/vagrant/ansible-prometheus/molecule/default/tests/...
Lint completed successfully.
--> Executing Ansible Lint on /home/vagrant/ansible-prometheus/molecule/default/playbook.yml...

Lint completed successfully.
--> Scenario: 'docker'
--> Action: 'destroy'

    PLAY [Destroy] *****************************************************************

    TASK [Destroy molecule instance(s)] ********************************************
    changed: [localhost] => (item=None)
    changed: [localhost]

    TASK [Wait for instance(s) deletion to complete] *******************************
    ok: [localhost] => (item=None)
    ok: [localhost]

    TASK [Delete docker network(s)] ************************************************

    PLAY RECAP *********************************************************************
    localhost                  : ok=2    changed=1    unreachable=0    failed=0

--> Scenario: 'docker'
--> Action: 'dependency'
Skipping, missing the requirements file.
--> Scenario: 'docker'
--> Action: 'syntax'

    playbook: /home/vagrant/ansible-prometheus/molecule/default/playbook.yml

--> Scenario: 'docker'
--> Action: 'create'

    PLAY [Create] ******************************************************************

    TASK [Create molecule instance(s)] *********************************************
    changed: [localhost] => (item=None)
    changed: [localhost]

    TASK [Wait for instance(s) creation to complete] *******************************
    changed: [localhost] => (item=None)
    changed: [localhost]

    PLAY RECAP *********************************************************************
    localhost                  : ok=2    changed=2    unreachable=0    failed=0

--> Scenario: 'docker'
--> Action: 'prepare'

    PLAY [Prepare] *****************************************************************

    PLAY RECAP *********************************************************************

--> Scenario: 'docker'
--> Action: 'converge'

    PLAY [Run role] ****************************************************************

    TASK [Gathering Facts] *********************************************************
    fatal: [docker]: UNREACHABLE! => {"changed": false, "msg": "Authentication or permission failure. In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in \"/tmp\". Failed command was: ( umask 77 && mkdir -p \"` echo ~/.ansible/tmp/ansible-tmp-1549558006.05-243433001157362 `\" && echo ansible-tmp-1549558006.05-243433001157362=\"` echo ~/.ansible/tmp/ansible-tmp-1549558006.05-243433001157362 `\" ), exited with result 1", "unreachable": true}

    NO MORE HOSTS LEFT *************************************************************

    PLAY RECAP *********************************************************************
    docker                     : ok=0    changed=0    unreachable=1    failed=0

ERROR:
An error occurred during the test sequence action: 'converge'. Cleaning up.
--> Scenario: 'docker'
--> Action: 'destroy'

    PLAY [Destroy] *****************************************************************

    TASK [Destroy molecule instance(s)] ********************************************
    changed: [localhost] => (item=None)
    changed: [localhost]

    TASK [Wait for instance(s) deletion to complete] *******************************
    changed: [localhost] => (item=None)
    changed: [localhost]

    TASK [Delete docker network(s)] ************************************************

    PLAY RECAP *********************************************************************
    localhost                  : ok=2    changed=2    unreachable=0    failed=0

When I look at the logs though, I see the container being ran (shim containerd-shim started) and destroyed (shim reaped). Then ansible try to get in the container but dockerd tells that the container is not running :
Feb 07 16:46:42 ubuntu-xenial containerd[12040]: time="2019-02-07T16:46:42.746708718Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/7c1b8ce99befde09676ee468959cf1e3869e051ea7feadc5a94d1609f1917e44/shim.sock" debug=false pid=17335
Feb 07 16:46:42 ubuntu-xenial kernel: eth0: renamed from vethd6e1338
Feb 07 16:46:42 ubuntu-xenial kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth497c3b5: link becomes ready
Feb 07 16:46:42 ubuntu-xenial kernel: docker0: port 1(veth497c3b5) entered forwarding state
Feb 07 16:46:42 ubuntu-xenial kernel: docker0: port 1(veth497c3b5) entered forwarding state
Feb 07 16:46:43 ubuntu-xenial ansible-async_wrapper.py[17293]: Module complete (17293)
Feb 07 16:46:43 ubuntu-xenial containerd[12040]: time="2019-02-07T16:46:43.069769009Z" level=info msg="shim reaped" id=7c1b8ce99befde09676ee468959cf1e3869e051ea7feadc5a94d1609f1917e44
Feb 07 16:46:43 ubuntu-xenial dockerd[12182]: time="2019-02-07T16:46:43.080701027Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Feb 07 16:46:43 ubuntu-xenial kernel: docker0: port 1(veth497c3b5) entered disabled state
Feb 07 16:46:43 ubuntu-xenial kernel: vethd6e1338: renamed from eth0
Feb 07 16:46:43 ubuntu-xenial kernel: docker0: port 1(veth497c3b5) entered disabled state
Feb 07 16:46:43 ubuntu-xenial kernel: device veth497c3b5 left promiscuous mode
Feb 07 16:46:43 ubuntu-xenial kernel: docker0: port 1(veth497c3b5) entered disabled state
Feb 07 16:46:44 ubuntu-xenial ansible-async_wrapper.py[17218]: Done in kid B.
Feb 07 16:46:46 ubuntu-xenial dockerd[12182]: time="2019-02-07T16:46:46.075583751Z" level=error msg="Error setting up exec command in container docker: Container 7c1b8ce99befde09676ee468959cf1e3869e051ea7feadc5a94d1609f1917e44 is not running"
Feb 07 16:46:46 ubuntu-xenial dockerd[12182]: time="2019-02-07T16:46:46.109063857Z" level=error msg="Error setting up exec command in container docker: Container 7c1b8ce99befde09676ee468959cf1e3869e051ea7feadc5a94d1609f1917e44 is not running"
Feb 07 16:46:46 ubuntu-xenial dockerd[12182]: time="2019-02-07T16:46:46.143361032Z" level=error msg="Error setting up exec command in container docker: Container 7c1b8ce99befde09676ee468959cf1e3869e051ea7feadc5a94d1609f1917e44 is not running"
Feb 07 16:46:47 ubuntu-xenial ansible-async_wrapper.py[17583]: Invoked with<PHONE_NUMBER>94 7200 /home/vagrant/.ansible/tmp/ansible-tmp-1549558007.23-50339111557339/AnsiballZ_docker_container.py _
Feb 07 16:46:47 ubuntu-xenial ansible-async_wrapper.py[17583]: Return async_wrapper task started.
Feb 07 16:46:47 ubuntu-xenial ansible-async_wrapper.py[17586]: Starting module and watcher
Feb 07 16:46:47 ubuntu-xenial ansible-async_wrapper.py[17586]: Start watching 17587 (7200)
Feb 07 16:46:47 ubuntu-xenial ansible-async_wrapper.py[17587]: Start module (17587)
Feb 07 16:46:47 ubuntu-xenial ansible-async_wrapper.py[17292]: Done in kid B.
Feb 07 16:46:47 ubuntu-xenial ansible-async_wrapper.py[17587]: Module complete (17587)
Feb 07 16:46:52 ubuntu-xenial ansible-async_wrapper.py[17586]: Done in kid B.

And in fact when ansible try to get in the container Container 7c1b8ce99befde09676ee468959cf1e3869e051ea7feadc5a94d1609f1917e44 just destroyed before, he fails.

I think my problem happening because I use docker:dind and the default behavior of this image is to be destroyed if nothing hang them up. I've dig up a bit in the other images used by molecule and they are all made to launch systemd. Once they're up they don't die and the converge step can go on
docker:dind uses alpine as a base so I  won't be able to use systemd. I'll try to find an image with supervisord or runit already built, if nothing came up, I'll setup one myself, and test my theory.

I've setup docker:dind to have supervisord as an entrypoint. It works but the role being compatible only with centos or debian, I have a fail at the very firest task of the role..
I'll have to find another base image :)

I've build localy an image using billyteves/ubuntu-dind:16.04. the issue I had is fixed. I'm still stuck on other tasks but I have made progress I hope to have something localy soon

I've built a docker-dind based on ubuntu with supervisord to mimic what's needed for molecule to test the role.
Its working up to the point where the container is suposed to be running. At that point, the container fails.
The thing is for the configuration file of prometheus, I'm doing a bind mount of the file to the container and it fails for molecule because all the initialisation of prometheus happen in a container. binds mounts though happens between the host and the prometheus container because the molecule container is built and build the prometheus container with the same dockerd engine that is on the host. Therefore when the container is starting, no config file is found :
    TASK [ansible-prometheus : ensure container is up] *****************************
    fatal: [docker]: FAILED! => {"changed": false, "msg": "Error starting container 8bfa98f0c0d57ea0afa6a84bcc3c24e353dcf83ce666be920385dd3029763296: 400 Client Error: Bad Request (\"{\"message\":\"OCI runtime create failed: container_linux.go:344: starting container process caused \\\"process_linux.go:424: container init caused \\\\\\\"rootfs_linux.go:58: mounting \\\\\\\\\\\\\\\"/srv/prometheus/config/prometheus.yml\\\\\\\\\\\\\\\" to rootfs \\\\\\\\\\\\\\\"/var/lib/docker/overlay2/efb5913f5cf582a7bb1e9ad101fe18660fb2340731a39c33471ab436a3a97db6/merged\\\\\\\\\\\\\\\" at \\\\\\\\\\\\\\\"/var/lib/docker/overlay2/efb5913f5cf582a7bb1e9ad101fe18660fb2340731a39c33471ab436a3a97db6/merged/etc/prometheus/prometheus.yml\\\\\\\\\\\\\\\" caused \\\\\\\\\\\\\\\"not a directory\\\\\\\\\\\\\\\"\\\\\\\"\\\": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type\"}\")"}

All other bind mounts will fail. The data volume was setup with a bind option to keep control of the location of prometheus data, but  if I let docker manage the volume, the container works with the volume, but the came can't be done with external files such as configuration files.
The role is working but I don't think that I can go much further on with molecule because of the way molecule and docker works with each other. A cleaner way would be to have a vagrant like driver for molecule to use instead of docker and work with that instead of docker containers.
Note : I've try to emphasis that he problem is happening with the docker engine, because of all the docker in docker happenning, things can be a bit confusing !

I've updated the documentation to explain usage of variables related to docker

The role is working but I don't think that I can go much further on with molecule because of the way molecule and docker works with each other. A cleaner way would be to have a vagrant like driver for molecule to use instead of docker and work with that instead of docker containers.

Note : I've try to emphasis that he problem is happening with the docker engine, because of all the docker in docker happenning, things can be a bit confusing !

Since vagrant cannot be run (without hacks) in travis CI, docker is the only option. Additionally if this cannot be tested in CI, then I cannot merge it as this project is just an after-hours one and I don't want to spend a lot of time chasing bugs.

Hi,
I've anwser some of your feedbacks, and read all of them carefully :)
Thank you so much for all the feedback given over this PR, it was a first on a lot of levels, for me and my team, we've appreciated every feedback !!
Unfortunatly as you noticed I can't provide a better solution regarding the test made to build the container. I agree with you that we can't proceed further on this PR as this is holding any QA over this role.
I think that we're stuck until travis support vagrant (or any similar alternative) and molecule support any vm providers (I did not saw moelcule having a libvirt or virtualbox provider as well).
So until that is an option, or someone is having a better solution / conception of this idea, I think we can close this PR.

I think you are right. Thanks for all the effort you put into creating this solution!


Print context of violations

Example:
warning: test.f90:90:3: P011 prefer 'real(real64)' to 'double precision' (see 'iso_fortran_env')
   |
89 |   ! Should trigger for use of 'double precision'
90 |   double precision function double_prec(x)
   |   ^ P011
91 |     double precision, intent(in) :: x
92 |     double_prec = 2 * x
   |

warning: test.f90:91:5: P011 prefer 'real(real64)' to 'double precision' (see 'iso_fortran_env')
   |
89 |   ! Should trigger for use of 'double precision'
90 |   double precision function double_prec(x)
91 |     double precision, intent(in) :: x
   |     ^ P011
92 |     double_prec = 2 * x
93 |   end function
   |

warning: test.f90:97:1: M001 subroutine not contained within (sub)module or program
   |
96 | ! This function should trigger for missing enclosing module
97 | subroutine triple(x)
   |  ^ M001
98 |   integer, intent(inout) :: x
99 |   x = x * 3
   |

A couple of annoying things:

Something really weird going on here, where I can't get it to
put the annotation in the first column: it's either in column 2
or the end of the previous line. But does appear to be right
for other columns!
Some annoyance here: we have to have some level prefix to our
message. Might be fixed in future version of annotate-snippets
-- or we use an earlier version with more control.
Also, we could use .origin(path) to get the filename and
line:col automatically, but see above about off-by-one error

As an aside, it might be nice to introduce a --concise mode that doesn't include the snippet, as it seems to be taking about 4 times as long when applied to a project I'm using for simple benchmarking (although 2 seconds to produce over 10000 annotated code snippets is still impressively fast as far as I'm concerned).


Modal with a parameter

Any idea on how I can show a modal and pass it content string in a parameter? Using the starter app which is great but this particular use case isn't included and I'm at a bit of a loss as to whether there's an elegant way to do it.

Hey I had asked this question earlier in the year. Do you think this helps your case?
https://github.com/kanzitelli/rn-navio/issues/19#issuecomment-1518709576

Sorry I didn't pick that up before, my bad. Thanks for the tip!
On Thu, Dec 21, 2023 at 6:57 AM Damian @.***> wrote:

Hey I had asked this question earlier in the year. Do you think this helps
your case?
#19 (comment)
https://github.com/kanzitelli/rn-navio/issues/19#issuecomment-1518709576
—
Reply to this email directly, view it on GitHub
https://github.com/kanzitelli/rn-navio/issues/28#issuecomment-1866200443,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AQRSUN7DV7FP73HS5X3423TYKQW25AVCNFSM6AAAAAA63LG5XKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRWGIYDANBUGM
.
You are receiving this because you authored the thread.Message ID:
@.***>

Hi! You can find the solutions under Passing params to a modal section. With Navio v0.1.0 you can use navio.modals.show('ModalName', { data }).


Audio + MIDI

Hi,
I need to create a JACK client that plays an audio file and a MIDI file at the same time.
I'm trying to merge this two examples:

https://github.com/spatialaudio/jackclient-python/blob/master/examples/midi_file_player.py
https://github.com/spatialaudio/jackclient-python/blob/master/examples/play_file.py

I miss the last step (I hope), i.e. merging the process() functions:

https://github.com/spatialaudio/jackclient-python/blob/master/examples/midi_file_player.py#L37
https://github.com/spatialaudio/jackclient-python/blob/master/examples/play_file.py#L64

Do you have some suggestions? If I understand correctly, this while loop blocks the script until the end of the MIDI file, but I have to play MIDI and audio in parallel.
Thank you!
Carlo

What do you think about replacing this def process(frames): with def midi_process(frames):, and this def process(frames): with def audio_process(frames):?
@client.set_process_callback
def midi_process(frames):
    [...]

@client.set_process_callback
def audio_process(frames):
     [...]

How can client.set_process_callback understand what kind of work (processing audio or MIDI) has to be done?

@C0nsultant has already answered your question about multiple callbacks: https://github.com/spatialaudio/jackclient-python/issues/101#issuecomment-726088725
Coming back to your original question, I would say it very much depends on how exactly you want to read your audio file and what requirements you have for that.
Would it be OK to read the whole audio file into memory at once?
This would make the callback much simpler, but it of course requires that the whole audio file fits into memory.

@mgeier,
thank you.

Would it be OK to read the whole audio file into memory at once?

Yes, no problem.
Here is my current (not working) code:
https://github.com/Plantarium-Seregno/Luces/blob/main/luces.py

Would it be OK to read the whole audio file into memory at once?

Yes, no problem.

In that case, you don't need threading.Thread and you shouldn't use it.
If both the audio and the MIDI file are loaded into memory in their entirety, you can access them directly in the process callback. This means you don't need the queue either.
In the process callback you can simply get the current audio and MIDI data and send it to the hardware.
It's really much simpler than your code suggests!
Here's my suggestion what you should do to make this work:

remove everything that mentions threading.Thread
remove everything that mentions queue in any form
concentrate on either audio or MIDI first, comment the other one out
once one of them works (and only then), try to add the other one back in

Once you've reached the third point (but not earlier), feel free to share your code here, then I can have a look.

Thank you very much, @mgeier!
I'm concentrating on the audio part; I'm trying to get a very minimal script from scratch.
#!/usr/bin/env python3
"""Controller for Luces.

This plays a MIDI file and an audio file at the same time.

"""

import jack
import soundfile as sf
#from mido import MidiFile

project_dir = 'Media'
project = 'test'
audio_file = '{}/{}.wav'.format(project_dir, project)
midi_file = '{}/{}.mid'.format(project_dir, project)

client = jack.Client('Luces')

def audio_process(frames):
    pass

def midi_process(frames):
    pass

@client.set_process_callback
def process(frames):
    try:
        audio_process(frames)
    except jack.CallbackExit:
        # proper handling recommended
        pass
    try:
        midi_process(frames)
    except jack.CallbackExit:
        # proper handling recommended
        pass

if __name__ == "__main__":
    audio_out = client.outports.register('audio_out') # Mono
    target_audio_port = client.get_ports(is_physical=True, is_input=True,
            is_audio=True)[0] # Audio Output 1 (L)

    client.activate()

    client.connect(audio_out, target_audio_port)

    with sf.SoundFile(audio_file) as f:
        pass

    audio_out.disconnect()

    audio_out.unregister()

    client.deactivate()
    client.close()

Now I can't understand how to use soundfile and audio_process(frames) in order to send audio data to the sound card: in your example there is a massive use of threading' and queue`, so it results very difficult for me...

Very good, we are getting closer ...
My next question is: Do you really need if __name__ == "__main__"?
If not, you can remove it.
If yes, you'll have to move the definition of the process callback into the if __name__ == "__main__" block.
Also: using client.activate() and then deactivate() and close() looks very much like it could need a with statement, right?
I guess the disconnect() and unregister() would happen automatically (right?), but it's of course OK to keep it there if you want to bring attention to the cleanup.
Also: be careful when using pass while handling those exceptions (see https://github.com/spatialaudio/jackclient-python/issues/101#issuecomment-726634601). You might want to consider using raise instead.
Or you remove the callback separation for now, to keep it simple.
You can split the callback in two parts later, if needed.

in your example there is a massive use of threading and queue, so it results very difficult for me...

I know, but the whole point why I asked whether you can load the whole file into memory was to be able to get rid of this difficult stuff.
You can load the entire audio file somewhere outside the callback function:
audio_data, samplerate = sf.read(audio_file)

The variable audio_data should then be visible in the callback function, where you can use NumPy indexing to get the right slice of it to play back.
I probably should make an example for this at some point, but for now you can probably start with something like:
outdata[:] = audio_data[current_frame:current_frame+blocksize]

You'll have to calculate the current_frame, of course.
Do you know the number of channels beforehand?
If not, you might have to be a bit more careful with indexing to make it work for a single-channel file.
Also, you'll have to consider what happens when the audio file is over.
After that, you'll have to stop playback or you'll have to fill the output with zeros.
And you should consider the situation where the very last block of the file is smaller than the blocksize used in the callback.
In this case you'll have to copy the last partial block and then fill the rest of the frames with zeros.

Thank you, @mgeier, for your kind reply.
Here is my current attempt:
#!/usr/bin/env python3
"""Controller for Luces.

This plays a MIDI file and an audio file at the same time.

"""

import jack
import soundfile as sf
#from mido import MidiFile

project_dir = 'Media'
project = 'test'

audio_file = f'{project_dir}/{project}.wav'
midi_file = f'{project_dir}/{project}.mid'

audio_data, samplerate = sf.read(audio_file)

client = jack.Client('Luces')

audio_out_port = client.outports.register('audio_out') # Mono
target_audio_port = client.get_ports(is_physical=True, is_input=True,
        is_audio=True)[0] # Soundcard Audio Output 1 (L)

current_frame = 0
out_audio_data = audio_data[current_frame:current_frame+client.blocksize]
audio_buffer = audio_out_port.get_array()

@client.set_process_callback # TODO: to be removed
def audio_process(frames):
    # TODO: How can I send out_audio_data to audio_out_port?
    #       The following code doesn't work (or it is uncomplete)
    if len(out_audio_data) < client.blocksize:                          # Last block
        out_audio_data += [0]*(client.blocksize-len(out_audio_data))    # Fill with zeros
    audio_buffer[:] = out_audio_data

    current_frame += client.blocksize
    out_audio_data[:] = audio_data[current_frame:current_frame+client.blocksize]

def midi_process(frames):
    pass

<EMAIL_ADDRESS># TODO: to be uncommented
def process(frames):
    try:
        audio_process(frames)
    except jack.CallbackExit:
        # proper handling recommended
        pass
    try:
        midi_process(frames)
    except jack.CallbackExit:
        # proper handling recommended
        pass

with client:
    client.connect(audio_out_port, target_audio_port)

# TODO: I have to stop the playback when the audio file is over.

There is something wrong: I have no errors, but the process terminate almost immediately.

Do you know the number of channels beforehand?

Yes, I do.
The workflow is: Ardour → Export to Mono Audio → Python Client → Soundcard Audio Output 1 (L)

Very good, it's getting simpler!

Do you know the number of channels beforehand?

Yes, I do.

OK, that's good, then you don't have to worry about channel indexing, because your arrays are all one-dimensional.

There is something wrong: I have no errors, but the process terminates almost immediately.

And then?
Nothing more to do, the script is finished and Python interpreter terminates.
You will somehow have to wait ... probably a threading.Event helps with that?
I think I've used that in some of the examples ...
But that's not the only problem. Another one is the call to get_array().
The docs (https://jackclient-python.readthedocs.io/en/0.5.3/api.html#jack.OwnPort.get_array) say:

This method shall only be called from within the process callback

Thank you, @mgeier!
https://github.com/Plantarium-Seregno/Luces/blob/main/luces.py
Now, after having fixed the remaining TODOs, I'd like to implement 3 new features:

adjustable delay for compensating the delay between audio and MIDI caused by the speed of sound
playlist: a list of projects the script must automatically execute
scheduling: 3 different scenarios for 3 moments of the day (fixed lights with no music during the night, no music and no lights during the daytime, and the musical light show at the evening)

What should pay attention to in order not to break the script?
My hypotheses:

using time.sleep (where and how?)
read a file containing the list of projects; how to run a new project when the playback is finished?
using schedule

Thank you!

I'm glad that it works now, that's great!
I don't think that time.sleep() is the right tool for the job, given that the audio/MIDI playback runs asynchronously.
If you call time.sleep() in the main thread, it will not change anything and playback will continue, if you call it in the audio thread, audio playback will break and you'll get xruns.

adjustable delay for compensating the delay between audio and lights

That very much depends on what you mean by "adjustable".
Is it enough to have a constant value (that you find by trial and error), or do you need to change the delay dynamically during playback?
The latter case could become quite complicated, so I hope the former is sufficient.
If that's the case, I think there is a quite simple way:

if audio is too early, insert a few zeros in the beginning of the array.
if MIDI is too early, start with an offset greater than zero. The value is in seconds, so this should be easy.

read a file containing the list of projects; how to run a new project when the playback is finished?

There are many ways to do this, here are just a few ideas:

Add command line parsing to your script and re-start your script with different parameters once it is finished. You could do that e.g. with a shell script, or with another Python script or whatever.
Refactor your code into a reusable function that accepts the the project parameters as function parameters. Then you can create a new Python script that calls your function repeatedly.
Interweave the playlist logic directly into your existing code

In the first one you'll have to create a new JACK client and connect to JACK each time, but I think with the other options you could reuse the same JACK client and keep it connected all the time.
I don't know what's best here, I guess it is mostly a matter of taste. And of course it depends on how exactly you are planning to run this whole thing.

Hello, @mgeier,
thank you for your replay and sorry for the delay.

* if MIDI is too early

This is my case: speed of light is much, much faster than speed of sound; so, at 50-100 m (or more) from the source you see the lights immediately, but the sound arrives after some time (depending on the distance).

start with an offset greater than zero. The value is in seconds, so this should be easy.

Unfortunately it doesn't work. "Normal" values are expressed in milliseconds; for example for a distance of 100 m, delay is (more or less) 300 ms, but I can't put offset = 0.3 because this value must be an integer.
Is there a solution?

* Add command line parsing to your script and re-start your script with different parameters once it is finished. You could do that e.g. with a shell script, or with another Python script or whatever.

I don't like this way, but I found it simpler, so at the moment I'm using a second Python script.

* Interweave the playlist logic directly into your existing code

In TODO list :)
Thank you again!

start with an offset greater than zero. The value is in seconds, so this should be easy.

Unfortunately it doesn't work. "Normal" values are expressed in milliseconds; for example for a distance of 100 m, delay is (more or less) 300 ms, but I can't put offset = 0.3 because this value must be an integer.

Oh, sorry, I was wrong. The offset value is in samples.
So if I'm not mistaken again, you can use offset = round(0.3 * samplerate) to get a delay of 300 ms.
BTW, you should make sure that JACK uses the same sample rate as your audio files!

So if I'm not mistaken again, you can use offset = round(0.3 * samplerate) to get a delay of 300 ms.

Yes, that works!
Thank you very much.

BTW, you should make sure that JACK uses the same sample rate as your audio files!

Nice to know, thanks.


large datasets - how to handle them?

i read docs couldn't really find anything clarifying this, although title in readme file pretty much says "in-memory"
suppose i have a 6gb structured dataset,
in pandas for instance i can specify chunksize what is equivalent idiomatic way in dataframe? ( i see readLine parameter in read methods but I'm not sure)

thanks

Hi! I think nobody tried it before, so there is no idiomatic way to do it right now. What's the format of your dataset, CSV?

part 1: description of the problem
in my case i'm using a postgres database as my initial dataset, i first saved result of a huge hierarchical query in disk using json_agg and  recursive query to end up with a huge json file which is saved to file using copy ... to syntax
i even later wrote this naive implementation so if i could find a way to stream data i can directly parse from sql resultset:
fun ResultSet.toDataFrame(): DataFrame<*>{
    val cols = with(metaData) {
        (1 .. columnCount)
            .map { getColumnName(it) }
            .toList()
    }

    val resultList = sequence {
        while (this@toDataFrame.next()) {
            val mapRepoOfObject = mutableMapOf<String,Any>()
            cols.forEach() {
                mapRepoOfObject[it] = getObject(it)
            }
            yield(mapRepoOfObject)
        }
    }.toList()

    return dataFrameOf(cols){ header ->
        resultList.map { it[header]!! }.toList()
    }
}

// use case

val con: Connection =  DriverManager.getConnection("jdbc:postgresql://blahblah/")!!
val df = con.createStatement().executeQuery("select * from huge").toDataFrame().also { println(it) }

if i use limit and pagination i can produce stream of dataframes,
with the same procedure i can also end up with the same stream of dataframes from file
part 2: so what's the problem?
for instance have groupby or any aggregate in mind, processing multiple dataframes in separate fashion can result in duplicates?
so basically i'm inspecting if we can use DataFrame for larger datasets some how

Hello again. I've found some interesting article on groupBy / aggregate for chunked dataframes in pandas https://maxhalford.github.io/blog/pandas-streaming-groupby/. The general idea is that if you need a groping by some characteristic, sorting a dataset by this characteristic could help to process your data in chunks and then merge the result

I have the same concern about large dataset but a different scenario and desired solution.
I am not too interested in manipulating the dataframe as a whole but rather reading row by row (step by step), in a lazy fashion (again because my data is huge files say 8GB, 20 GB). The formats I am interested in are XML, CSV

@corlaez some operations by nature require traverse of whole dataset (like transformation, count , etc ...) and some operations even require auxiliary storage (like grouping, distinct, ... )
the answer i've found is  dataframe is for researching and testing on a small subset of data
for anything extra we should use the specific tool made and optimized to be used as a pipeline on big data (like sparks and hadoop (i'm not qualified to give advice on this ))
as the result i like dataframe as is and i don't expect it to grow complex to support such usecases
although it would be very beneficial if dataframe team decide to publish extra modules to interpolate with those solutions to have a seamless experience.

It was also asked here, on the Reddit https://www.reddit.com/r/Kotlin/s/ctNLZEq2gI
Also it should be provided a documentation page for user with different option and ways, how to handle large CSV files, what could be fine-tuned
Benchmark for some synthetic CSV is also worth to add


Registry explore cmd

What was wrong?
Issue #88
How was it fixed?
Added a new subparser to registry that accepts a valid URI and prints the total releases with their corresponding version and manifest_uri as well as the package name.
Cute Animal Picture

I'll push a fix for the linter if the formatting looks good on the output and add the corresponding tests!

Should have the tests ready this weekend

@corydickson No worries! just ping me whenever this is ready for a review

@njgheorghita Thank you for the patience, should be ready for review!

May I make the changes suggested by the linter here?

Also the mypy fixes are in master now if you want to rebase against master to fix the linting errors.

lol sorry for the fp was having issues... should be rebased correctly

Looks good, thanks @corydickson !!


make all chat area components tabbable (accessibility)

most of the elements i the chat area are either tabable but missing focus visualization, or are completely left out of the tab flow. a few other minor issues also resolved. see
Before
https://user-images.githubusercontent.com/5056972/228095847-f2e636fb-49f2-4c90-9286-1c54459c7e2e.mp4
https://user-images.githubusercontent.com/5056972/228095869-2c3aea55-515a-4892-a656-c4de6cbafbe3.mp4
After:
https://user-images.githubusercontent.com/5056972/228096558-b3c81c36-0e28-4239-9f58-355493977cd9.mp4
https://user-images.githubusercontent.com/5056972/228096570-81b119d3-6d4d-4562-99be-95636d03c4dc.mp4
fixes: #245

You are on a roll.
Really appreciate the comments here. I learned a lot from this one.


[Feature Request] Page to administrate roles

Problem description
Managing roles has not been implemented yet in the researcher UI. However, one should be able to create/edit/view/delete roles (if they have proper permissions).
Desired solution
Page with role management in administration
Additional context
The old UI already has this implemented - go there for inspiration

#175 includes the main implementation of this issue.
Separate smaller issues have been created to implement the last few bits: #186 #185 #184 #183


[FEATURE] GetAlerts API

Is your feature request related to a problem?
SAP needs API to fetch all Alerts for a Detector
What solution would you like?
Add GetAlerts API to fetch all Alerts related to a Detector
REQUEST:
Params:

param
description

detectorId
ID of Detector to fetch alertings for

severityLevel  (OPTIONAL)
Used to filter by Alert SeverityLevel

alertState. (OPTIONAL)
Used to filter by Alert State. Possible values: ACTIVE, ACKNOWLEDGED, COMPLETED, ERROR, DELETED

sortString. (OPTIONAL)
This field specifies which string the Alerting plugin uses to sort the alerts

sortOrder. (OPTIONAL)
The order to sort the list of findings, either ascending or descending

missing. (OPTIONAL)
list of fields in given index for which there are no found alias mappings

size. (OPTIONAL)
An optional limit for the maximum number of results returned in the response

startIndex. (OPTIONAL)
The pagination indicator

searchString. (OPTIONAL)
The alert attribute you want returned in the search

GET /_plugins/_security_analytics/alerts?detectorId=detector_1234

@petardz plz add a sample response structure.

@petardz @eirsep What are possible values for state in the response other than "ACTIVE"?

Would it be same as https://opensearch.org/docs/latest/monitoring-plugins/alerting/monitors/#work-with-alerts

@amsiglan Yes, same as on link you posted. Values are upper-case


Added Chat to the Example Sync Server and Client

Added Chat example to server/client sync example
Updated Netty to 4.1.52.Final
Cleaned up gradle build files
Updated Proguard config
Updated SSLHandler import of keys to look for Sync.key and Sync.crt in the resources folder
Updated SSL documentation in readme
Updated TICK documentation in readme
Removed PASSTHROUGH mode due to Netty deadlock bug caused by writing in ChannelInputHandler read

🍌


update bitrise.yml and step.sh

bitrise.yml changes:

Switch working dir step makes sure that your step does not depends on the current working directory
Using path::./: you tells bitrise-cli to use the step in the current directory, so it will use the current local state of your step
My previous suggestion about wetransfert installation was incorrect, sorry about that. wetransfert has to be installed by: npm install wetransfert --save to make the step able to use the wetransfert node module

Ok i understand thanks.

I just add @godrei --prefix $THIS_SCRIPT_DIR to install the package in the right directory.
I test and send a new PR ;) thanks a lot @godrei


Allow to automatically trim the PoolThreadCache in a timely interval

Motivation:
PooledByteBufAllocator uses a PoolThreadCache per Thread that allocates / deallocates to minimize the performance overhead. This PoolThreadCache is trimmed after X allocations to free up buffers that are not allocated for a long time. This works out quite well when the app continues to allocate but fails if the app stops to allocate frequently (for whatever reason) and so a lot of memory is wasted and not given back to the arena / freed.
Modifications:

Add a ThreadExecutorMap that offers multiple methods that wrap Runnable / ThreadFactory / Executor and allow to call ThreadExecutorMap.currentEventExecutor() to get the current executing EventExecutor for the calling Thread.
Use these methods in the constructors of our EventExecutor implementations (which also covers the EventLoop implementations)
Add io.netty.allocator.cacheTrimIntervalMillis system property which can be used to specify a fixed rate / interval on which we should try to trim the PoolThreadCache for a EventExecutor that allocates.
Add PooledByteBufAllocator.trimCurrentThreadCache() to allow the user to trim the cache of the calling thread manually.
Add testcases
Introduce FastThreadLocal.getIfExists()

Result:
Allow to better / more frequently trim PoolThreadCache and so give back memory to the area / system.

awesome @normanmaurer
I will take a deeper look as soon as possible

Just rebased it.


[Bug]: Mobile- CTA buttons are clicked/toggled when choosing an LLM provider in custom setup

Is there an existing issue for the same bug?

[X] I have checked the existing issues.

Describe the bug and reproduction steps
In the custom setup flow:

Launch the app on mobile
Click on the LLM provider input to select
Click on an option that directly overlays the “Save” button
The button becomes enabled.
Same flow for “Advanced option toggle” and “Close button”

P.s: it may be difficult to reproduce but have a look at the screen record shared to gain more context.
OpenHands Installation
app.all-hands.dev
OpenHands Version
No response
Operating System
None
Logs, Errors, Screenshots, and Additional Context
https://github.com/user-attachments/assets/f252b838-70e6-4273-8885-3073e4d6b6b6

Thanks for opening up this issue. We haven't really focused on how OpenHands works on mobile in general so there's likely a lot of bugs there.


Jackson 2.9.7 -> 2.9.8

Jackson 2.9.8 release notes:
https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.9.8
Jackson 2.9.8 improves security ("Block more classes from polymorphic deserialization")


Update to ssi v0.6.0

Update ssi version to v0.6.0 from v0.4.0

[x] Initial updates complete passing tests
[ ] Check DID publishing and attestation functionality retained

Update operations currently fail with published crate due to ssi-jwt::encode_sign() adding a type field to the JWT header. At first glance, this seems at odds with ION and also returns the following error when attempting to publish an update with the ION test node:
{"code":"jws_protected_header_missing_or_unknown_property","message":"jws_protected_header_missing_or_unknown_property"}

Current workaround is to remove the type_ that from the encode_sign method in our fork and update deps to point to this. This modification has been tested and successfully publishes an attestation.


Bug: Editor UI is fullscreen

Sadly the Component Selection in the Editor is completly fullscreen which makes it rly hard to edit other components.

Hey! I think you can just drag the component tray from the right to the left to resize it! This happens because I have not set a max-size.

Related issue #9

hey thank you so much for your help! gonna try this :)
Thank you for your work btw


Send  specific user agent to track usage

https://github.com/bundler/bundler/blob/master/lib/bundler/fetcher.rb#L140-L169

This seems worthwhile and easy enough to get in on the first release

Done done done


Cgroups v2 support for cgroup-limits

Fixes #201

I've tried to address all the issues mentioned above inline, and added a test. A problem with the test and cgroups-limits generally is that not everything works when running podman as non-root. In that case, the test does not run all test cases.

Although I am not cgroups expert, your changes look good to me. I'd add one more test for limited cpus.

Good idea, the last commit adds a simple scenario test for NUMBER_OF_CORES value, but only uses 1 processor set, to make the test pass on even single-core VMs.

Fedora tests fail for an unrelated reason. The changes looks good to me.


V1

Codecov Report

Merging #91 (367468c) into master (74bdf6b) will decrease coverage by 5.18%.
The diff coverage is 77.77%.

@@            Coverage Diff             @@
##           master      #91      +/-   ##
==========================================
- Coverage   90.56%   85.37%   -5.19%
==========================================
  Files          11       12       +1
  Lines         212      212
  Branches       41       41
==========================================
- Hits          192      181      -11
- Misses         20       31      +11

Impacted Files
Coverage Δ

src/modules/builder.ts
84.00% <ø> (ø)

src/create-module.ts
42.85% <42.85%> (ø)

src/app/app.ts
96.36% <100.00%> (-2.08%)
:arrow_down:

src/app/index.ts
100.00% <100.00%> (ø)

src/decorators.ts
81.81% <100.00%> (ø)

src/index.ts
100.00% <100.00%> (ø)

src/registry.ts
100.00% <100.00%> (ø)

src/modules/container.ts
80.95% <0.00%> (-9.53%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 74bdf6b...367468c. Read the comment docs.


Update Ubuntu and Debian instructions to use new repository

We now have a repository that only contains a single default
distribution, because the package is generic enough to works across all
recent Ubuntu and Debian releases.
The new repository includes old versions in the Packages index to
allow users to install old versions via apt install crate=x.y.z
It also contains ARM64 packages.

Thanks for updating the documentation! I just came here after looking for exactly this in connection with an ongoing support issue. Can we proceed with merging this PR?


[offsiteReq] Add DeepStore as a Prismic venue

DeepStore working days are Monday-Friday
Public holidays are non-working days

The logic as I understand it is:

10 DeepStore working days to get the item back on site
After that the item can be viewed any day that the library is open

If I make a request on Monday 3rd June:

the item will be delivered on site on Friday 14th June
it can be viewed Saturday 15th June, then Monday 17th to Saturday 22nd and Monday 24th to Friday 28th (the current choice of 12 days to pick from)
If there were x public holidays between Monday 3rd June and Friday 14th June, the delivery date would be postponed by as many days
If there were x public holidays between Saturday 15th June and Friday 28th we would add as many days after the 28th so that the user still has 12 possible days to come and view the item

That's assuming, TBC, that:

the day of the request (Monday 3rd June) is day 1 of the 10 days DeepStore lead time, not day 0
the item cannot be viewed on the day it is delivered, but the next day the library is open
we give the user a 12 opening day window to come on site to view the item once it's been delivered. If they want to see it at a later date, they ned to make the request at a later date

Email sent to LE&E to confirm the following assumptions:

the day of the request (Monday 3rd June) is day 1 of the 10 days DeepStore lead time, not day 0
the item cannot be viewed on the day it is delivered, but the next day the library is open
we give the user a 12 opening day window to come on site to view the item once it's been delivered. If they want to see it at a later date, they ned to make the request at a later date


.NET Coding Pack fails to install the VS extension

I have downloaded the '.NET Coding Pack for VS Code' (https://code.visualstudio.com/learn/educators/installers#_net-coding-pack) installer. VS Code (step 1) and the .NET SDK (step 2) get installed correctly. However, the third step, which is supposed to install the '.NET Extension Pack' extension into the newly installed VS Code, fails. The error message simply says: "Failed to install .NET Extension Pack" (see screenshot).
It looks like you have to launch VS Code once, so that it registers itself with Windows as the default program for VS Code extension packages. After that you can at least proceed manually.
Obviously, the .NET Coding Pack is meant for new .NET users to get started. Due to the above problem that turns out a pretty bad experience ...

Hey @profix898, thanks for letting us know about this. Since this seems to be an issue with the .NET extension pack, I've opened it in their repo: https://github.com/dotnet/installer/issues/11977


Stuck

https://paste.atlauncher.com/view/5394cea2-6e44-4085-bd48-a44828c6b1f1

The logs you have provided do not contain any useful information for analysis.

scroll to very bottom

At the bottom it's just world generation, without any useful exceptions to report errors.

it is not crash, it just stuck on loading so I killed the program

But this is a problem for which I cannot carry out a causal analysis.


Changes mdx-loader to check for prism language before trying to dynamically require the corresponding language file.

Currently, if someone tries to do a code block in a markdown file with an unknown language to prism, like so...
# my-markdown-file

```x
var i = 0;
```

...mdxc-loader will be unhappy and will throw an error trying to load the corresponding language file.
Module build failed: Error: Cannot find module 'prismjs/components/prism-x.js'

You can see the error happing on the MDXC demo site by removing the language from a code block.

The error can be avoided if the dynamic require is tucked into the check of available prism languages.

Thanks!


Custom Remotes in Browser VS Code

I'm trying to implement a custom remote for the browser version of VS Code. I'm trying to use the proposal resolvers API, so I'm targeting the Insiders build.
First I looked at vscode-remote-oss extension which does this for the desktop version. I installed the correct version of VS Code Server on my remote machine, and got my extension working on desktop by copy-paste-editing vscode-remote-oss. When I tried running this extension with vscode-test-web, I got an error saying:
ENOPRO: No file system provider found for resource 'vscode-remote://remote-oss+hostname/project/.vscode'

Then I looked at the vscode-test-resolver extension provided in this repository. I realized that the browser variant uses ManagedResolvedAuthority instead of ResolvedAuthority, thinking that only WebSocket-based remotes are supported in the browser. I converted my extension code to use WebSockets with ManagedResolvedAuthority instead of returning a ResolvedAuthority, but got the same error as above.
It seems that vscode-remote schema doesn't have a registered file system provider, so I tried registering one with vscode.workspace.registerFileSystemProvider('vscode-remote', memFs, { isCaseSensitive: true }) where memFs is the in-memory file system extension from sample extensions repository.
I didn't find any complete examples of remotes for web VS Code. Please reference ones that you are aware of. How should I go about this and do you have any pointers?

We closed this issue because we don't plan to address it in the foreseeable future. If you disagree and feel that this issue is crucial: we are happy to listen and to reconsider.
If you wonder what we are up to, please see our roadmap and issue reporting guidelines.
Thanks for your understanding, and happy coding!


LC0068 does not handle fully qualified table names

Hi
Thank you for your great work on LinterCop.
If you use fully qualified table names for tabledata in your Permissions property, LC0068 will give you a false positive: '...info LC0068: The current object is missing permission "x" for tabledata "y"'
It works perfectly if you use the "using" keyword for the namespace instead of using the fully qualified object name.
Expected:
As fully qualified object names are valid, I would expect LC0068 to also handle it correctly :-).

Thanks, I missed that.

Please retest with the most recent pre-release version

I can confirm it works here now (with latest pre-release version).
Thank you for your great support and for this fantastic Code Cop :-)


Apply transform to crop to keep in place

Should a transform be applied to cropped output to keep them in place with the original? Maybe this would be an option?

You mean something like layer.translation = [crop_min_z, crop_min_y, crop_min_x], right?

yeah exactly -- so the cropped image stays in place in the viewer. In fact, maybe it should be the default behavior.


[flutter_tools] Refresh VM state before executing hot reload

The VM's cached collection of isolates may contain stale data such as
references to isolates that have exited.  It should be refreshed
before hot reload sends reloadSources requests to each isolate.
Fixes https://github.com/flutter/flutter/issues/53094

Maybe there is a case that could be added to https://github.com/flutter/flutter/blob/master/packages/flutter_tools/test/integration.shard/background_isolate_test.dart
Something that spawns an isolate after some interaction/delay?

Does this fix the case where streams are not closed and subscriptions aren't cancelled?


.tar is empty

When I use webm format it works fine and downloads the webm, but if I use .png or jpg, the tar file that is downloaded is empty. I've included the tar.js.. there aren;t any errors in the console.
Why?

I am having the same issue @spite .