The Camera is still alive in background if the page pop while camera is starting

To reproduce :

Launch app code bellow
Tap the blue button to open the camera page.
While page is opening, clicking as soon as possible to the back button from appbar or your phone.

Behavior :
You will see the notification green badge which indicate that the camera is running in background despite the page is well poped (so high battery usage).
Expected behavior :
Green toast should not appear or disappear automatically. Camera should not live if controller is well disposed.
Video :
untitled.webm
Code :
import 'package:flutter/material.dart';
import 'package:mobile_scanner/mobile_scanner.dart';

void main() => runApp(const MaterialApp(home: MyHome()));

class MyHome extends StatelessWidget {
  const MyHome({super.key});

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(title: const Text('Flutter Demo Home Page')),
      body: SizedBox(
        width: MediaQuery.of(context).size.width,
        height: MediaQuery.of(context).size.height,
        child: Column(
          mainAxisAlignment: MainAxisAlignment.center,
          children: <Widget>[
            ElevatedButton(
              onPressed: () {
                Navigator.of(context).push(
                  MaterialPageRoute<dynamic>(
                    builder: (BuildContext context) =>
                    const BarcodeScannerWithController(),
                  ),
                );
              },
              child: const Text('MobileScanner with Controller'),
            ),
          ],
        ),
      ),
    );
  }
}

class BarcodeScannerWithController extends StatefulWidget {
  const BarcodeScannerWithController({super.key});

  @override
  _BarcodeScannerWithControllerState createState() =>
      _BarcodeScannerWithControllerState();
}

class _BarcodeScannerWithControllerState
    extends State<BarcodeScannerWithController>
    with SingleTickerProviderStateMixin {
  BarcodeCapture? barcode;

  final MobileScannerController controller = MobileScannerController(
    torchEnabled: true,
    autoStart: true,
    // formats: [BarcodeFormat.qrCode]
    // facing: CameraFacing.front,
    // detectionSpeed: DetectionSpeed.normal
    // detectionTimeoutMs: 1000,
    // returnImage: false,
  );

  bool isStarted = true;

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text(controller.hashCode.toString()),
      ),
      backgroundColor: Colors.black,
      floatingActionButton: FloatingActionButton(
        child: const Icon(Icons.camera),
        onPressed: () async {
          await controller.stop();
          await Navigator.of(context).push(
            MaterialPageRoute<dynamic>(
              builder: (BuildContext context) =>
              const BarcodeScannerWithController(),
            ),
          );
          await controller.start();
        },
      ),
      body: Builder(
        builder: (BuildContext context) {
          return Stack(
            children: <Widget>[
              MobileScanner(
                startDelay: true,
                controller: controller,
                errorBuilder: (
                    BuildContext context,
                    MobileScannerException error,
                    Widget? child,
                    ) {
                  return ScannerErrorWidget(error: error);
                },
                fit: BoxFit.contain,
                onDetect: (BarcodeCapture barcode) {
                  setState(() {
                    this.barcode = barcode;
                  });
                },
              ),
            ],
          );
        },
      ),
    );
  }

  @override
  void dispose() {
    controller.dispose();
    super.dispose();
  }
}

class ScannerErrorWidget extends StatelessWidget {
  const ScannerErrorWidget({super.key, required this.error});

  final MobileScannerException error;

  @override
  Widget build(BuildContext context) {
    String errorMessage;

    switch (error.errorCode) {
      case MobileScannerErrorCode.controllerUninitialized:
        errorMessage = 'Controller not ready.';
        break;
      case MobileScannerErrorCode.permissionDenied:
        errorMessage = 'Permission denied';
        break;
      default:
        errorMessage = 'Generic Error';
        break;
    }

    return ColoredBox(
      color: Colors.black,
      child: Center(
        child: Column(
          mainAxisSize: MainAxisSize.min,
          children: [
            const Padding(
              padding: EdgeInsets.only(bottom: 16),
              child: Icon(Icons.error, color: Colors.white),
            ),
            Text(
              errorMessage,
              style: const TextStyle(color: Colors.white),
            ),
          ],
        ),
      ),
    );
  }
}

Flutter doctor -v
[✓] Flutter (Channel stable, 3.3.9, on macOS 13.1 22C65 darwin-arm, locale fr-FR)
    • Flutter version 3.3.9 on channel stable at /Users/earminjon/fvm/versions/3.3.9
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision b8f7f1f986 (3 months ago), 2022-11-23 06:43:51 +0900
    • Engine revision 8f2221fbef
    • Dart version 2.18.5
    • DevTools version 2.15.0

[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.2)
    • Android SDK at /Users/earminjon/Library/Android/Sdk
    • Platform android-33, build-tools 33.0.2
    • ANDROID_HOME = /Users/earminjon/Library/Android/Sdk
    • Java binary at: /Users/earminjon/Library/Java/JavaVirtualMachines/corretto-11.0.17/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment Corretto-<IP_ADDRESS>.1 (build 11.0.17+8-LTS)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 14.2)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 14C18
    • CocoaPods version 1.11.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[!] Android Studio (version 2022.1)
    • Android Studio at /Users/earminjon/Library/Application Support/JetBrains/Toolbox/apps/AndroidStudio/ch-0/221.6008.13.2211.9514443/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    ✗ Unable to find bundled Java version.
    • Try updating or re-installing Android Studio.

[✓] IntelliJ IDEA Ultimate Edition (version 2022.3.2)
    • IntelliJ at /Users/earminjon/Applications/JetBrains Toolbox/IntelliJ IDEA Ultimate.app
    • Flutter plugin version 72.1.4
    • Dart plugin version 223.8617.8

[✓] IntelliJ IDEA Ultimate Edition (version 2022.3.2)
    • IntelliJ at /Users/earminjon/Library/Application Support/JetBrains/Toolbox/apps/IDEA-U/ch-0/223.8617.56/IntelliJ IDEA.app
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart

[✓] Connected device (3 available)
    • sdk gphone64 arm64 (mobile) • emulator-5554 • android-arm64  • Android 13 (API 33) (emulator)
    • macOS (desktop)             • macos         • darwin-arm64   • macOS 13.1 22C65 darwin-arm
    • Chrome (web)                • chrome        • web-javascript • Google Chrome 109.0.5414.119

[✓] HTTP Host Availability
    • All required HTTP hosts are available

! Doctor found issues in 1 category.

Maybe one fixed it will help following issue :

https://github.com/juliansteenbakker/mobile_scanner/issues/493
https://github.com/juliansteenbakker/mobile_scanner/issues/446 (issue close, but issue still reproductible)

I am having the same issue. Has anyone found a fix?

I am having the same issue. Has anyone found a fix?

Issue created 2 days ago :')
Update original post. If someone know how to resolve this, he will post don't worry !

the same issue

me too

Please check fix here
https://github.com/juliansteenbakker/mobile_scanner/issues/619#issuecomment-1918405545

@EArminjon With the new beta release, version 5.0.0-beta.1 , we removed the manual lifecycle management in mobile_scanner itself, as it caused issues with widget tree reparenting.
Now that you have full control over the lifecycle, this issue should no longer occur?

With latest version i can well manage with some tricks multiples scanners lifecycle (https://github.com/juliansteenbakker/mobile_scanner/issues/1002#issuecomment-2156693406) and so this issue.
As it's 'tricky' and required an other package 'visibility_detector' i didn't know if i can close this issue :/

When you leave the page, does that page call its dispose method?
You also do not await the disposal?
  @override
  Future<void> dispose() async {
    super.dispose(); // If dispose is async, the super call has to happen before any awaits. Otherwise the framework asserts
    await controller.dispose();
  }


Error while running the docker image

This docker image is great to seamlessly the algorithm trading process. Thanks for making it so easy!
I just encounter the following error when pull the image and run with the paper mode:
[2023-07-20 19:17:50,432]ERROR:API connection failed: ConnectionRefusedError(111, "Connect call failed ('<IP_ADDRESS>', 4001)")
[2023-07-20 19:17:50,432]ERROR:Make sure API port on TWS/IBG is open

I didn't see anywhere in the code we start the IBG explicitly, wondering any configure I'm missing?

+1 on this


this in-app purchase has been previously purchased

Version of flutter_inapp_purchase:  2.0.0+6
Platforms you faced the error (IOS or Android or both?):  ios
Expected behavior : one-time payment
Actual behavior : this in-app purchase has been previously purchased
Tested environment (Emulator? Real Device?):   Real Device?
Steps to reproduce the behavior : couldn't test in-app purchases

If your purchase is not consumable this is definitely expected!


trying to get LVGL8.X + LOVYANGFX + SD card with ESP32 S3 up and running ( VSC-PlatformIO-Arduino )

Hi,
I'm trying to get all together up and running.
Bord:  ESP32 S3 N16R8
Display: 3.5"TFT 320x480 V2.2 /  ILI9488 FT6236 / SD card socket
LovyanGFX version=1.1.12
lvgl version=8.3.11
Environment: VSC-PlatformIO-Arduino ( latest version )
Hardware connections ( PCB, soldered, no breadboard )
CS_LCD = GPIO_10
  cfg.pin_sclk = 12;            // SPIのSCLK
  cfg.pin_mosi = 11;            // SPIのMOSI
  cfg.pin_miso = 13;            // SPIのMISO
  cfg.pin_dc   = 41;            // SPIのD/C (-1 = disable)

  cfg.i2c_port = 1;
  cfg.i2c_addr = 0x38;
  cfg.pin_sda  = 38;
  cfg.pin_scl  = 39;
  cfg.pin_cs   = -1;
  cfg.freq     = 200000;

  cfg.bus_shared       =  true;

CS_SD card = GPIO_47
Status:
LCD and touch are working fine.
SD card is connected on same SPI, CS is separated on GPIO 47.
With oscilloscope ( THS730A ) I can see activities on GPIO 47, and SPI bus.
But it fails to mount the SD card ( 3.3V power is there, can't  find any diode in CS_LCD, so I expect, there is no conflict with MISO.
I tried out a 3 wire SPI to LCD without MISO connected to LCD, LCD still works fine, SD card doesn't work.
Think there is something else wrong and I spent already x nights without any success.
If I use the SD card alone in minimized testcase with SdFat.h, the card works fine ( same SPI connections, without LCD connected ).. BTW, finally I have to use exFAT with 64GB SD card.
Question: Is there an existing example / template available for VSC-PlatformIO-Arduino and ESP32 S3?
May be it is also a good solution, to use SPI3 for the SD card instead shared SPI ....
Thanks in advance for any useful hint

If SPI bus is same for display & sd card, configure cs pin for LovyanGFX and set shared to true.
After that use sd card without again initialising the bus.
Demo code is there in my ESP32-TUX project.

Thank you very much for fast response on sunday :-)
shared is set to true and I can see ( oscilloscope ) that the CS do not overlap.
But MiSO is on low, all the time.
Pull-up ( 2.2k ) did not help.

Hi,
still hanging,
think there is something missing.
CS, SCK and MOSI are visible at the SD card. MISO is low. But the SD card works fine with reduced SdFat example.
Format is exFAT, but other SD card with FAT32 doesn't respond also.
my setup()
....
....
lv_init();               // Bedienoberfläche initialisieren
Serial.println("lv_init erledigt");
lv_disp_draw_buf_init( &draw_buf, buf, NULL, screenWidth * 10 );
static lv_disp_drv_t disp_drv;
lv_disp_drv_init(&disp_drv);
Serial.println("Display Init erledigt");
// ********* Bildschirmparameter übergeben ********
disp_drv.hor_res = screenWidth;
disp_drv.ver_res = screenHeight;
disp_drv.flush_cb = my_disp_flush;
disp_drv.draw_buf = &draw_buf;
lv_disp_drv_register(&disp_drv);
Serial.println("Display Parameter gesetzt");
// ********* Initialisierung vom (dummy) input Device Treiber
static lv_indev_drv_t indev_drv;
lv_indev_drv_init(&indev_drv);
indev_drv.type = LV_INDEV_TYPE_POINTER;
indev_drv.read_cb = my_touchpad_read;
lv_indev_drv_register(&indev_drv);
gpio_pullup_en(GPIO_NUM_47);
if (tft.getStartCount() > 0)
{   // Free the bus before accessing the SD card
tft.endWrite();
}
// LCD + touch are working fine
SD.end();
delay(1000);

if ( SD.begin(SDCARD_SS_PIN, 2500000))
     // Something to manipulate the SD card.
 {
  Serial.println("Mount SD ok");
 }
else
 {
  Serial.println ("Mount SD not ok");
 }
lv_fs_open(file, "/file", LV_FS_MODE_WR);
if ( file )
 {
  Serial.println("Open file ok");
  lv_fs_close(file);
 }
 else
  {
   Serial.println("cannot open file");
  }

Your LovyanGFX config show’s display CS pin set to -1.

@Pepito-design Just curious. Did changing the CS pin work?


Cannot handle file name which starts with dot

Thanks for the work on this library. I am running into a bug today, the checker cannot handle file which starts with '.', for example, .gitignore and .npmrc. After a look at the source code, I found the root cause. In index.js, it uses path.extname to extract the file format. If filename starts with a dot, the extracting result will be empty.

Hi, are you able to tell me which version you are running? I have just tried this on a repository I use this package on and can confirm that it is checking the licenses of file beginning with a dot (I tried with an npmrc file)

Closed as unable to replicate


Fix wrong partitionsSpec type names in the document

Description
The doc for partitionsSpecs was changed in https://github.com/apache/incubator-druid/pull/8141, but it's using wrong type names for them. This PR fixes those wrong names.
Also, this PR adds some deprecated JSON properties in indexTuningConfig for backward compatibility.

This PR has:

[x] been self-reviewed.
[x] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
[x] added unit tests or modified existing tests to cover new code paths.

@clintropolis thanks for taking a look. Fixed conflicts.


Prometheus data from statistics

Statistics package should be able to deliver prometheus data

Make statistics use instead of a Counter?
https://prometheus.io/docs/guides/go-application/

All stats are not allowed to be exported with a boolean flag in the config.
Its a OPT in.


Automatically select constructor

Hi, i am using Wire(hyperion) and trying use Ceras for my library as main serializer because hyperion is poorly maintained and messagepack is not user friendly for customers:)
There is my model https://gist.github.com/DaniilSokolyuk/64f933c5fe5deb71f0c1cc35d59cf820, i have 2 problems

how to automatically select constructor like Wire
when i configured constructor by ....ConfigType<...>().ConstructBy(() => new MethodModel(null, ...)); and got an error "no source field or property could be found to populate the parameter 'Type declaringType'" but name is matched

Hi,
there's one thing I want to mention first; in case you only need that class you linked because you "manually" transmit MethodInfos: don't! 😄
Ceras already serializes things like FieldInfo, PropertyInfo, Type (well, obviously since that's a fundamental requirement of supporting interfaces / inheritance :P), and also MethodInfo.
Anyway, now as for your questions:

1. how to automatically select constructor like Wire

I'm not sure. How does Wire do it?
There are 3 major ways you can serialize something that has a constructor:
- Adding a constructor without parameters. If there is a ctor with no parameters (must not be private) Ceras will use it.
- Using TypeConfig to select one (as you did)

- Using an uninitialized object.

https://github.com/rikimaru0345/Ceras/blob/1dd0872cf61872f6fb941e7350eed6b4b5dbab22/src/Ceras/Config/TypeConfig.cs#L357-L360
But keep in mind that this does not call ANY constructor!! Meaning that if your constructor(s) don't just set fields/props, but also cause side effects; then those will not happen. However you can just use the [OnAfterDeserializeAttribute] to run any method after all members have been set 😄
That said; maybe Ceras should try to be smart and automatically select some constructor? Maybe filter out the ones that it couldn't possibly use (those where it can't match up members and parameters automatically), and then take the one with the most amount of parameters? And finally, if that results in multiple potential constructors, just throw.
Hmm, but then again, in pretty much every case where software tries to be smart it just causes more trouble than it's worth in the end 😂
What do you think?

when i configured constructor by...

That does indeed seem like a bug! 😮
How can I replicate it? Can you post some code? The setup of your SerializerConfig etc.

Thanks for the detailed answer!

Ceras already serializes things like FieldInfo, PropertyInfo, Type (well, obviously since that's a fundamental requirement of supporting interfaces / inheritance :P), and also MethodInfo.

Cool, but i also support the other serializers, so it was easier), and I also think MethodInfo stores a lot of garbage 😄

How does Wire do it?

but looks like by uninitialized object too, in my test for MethodModel, constructor is not called щт deserialize :(, sorry, i will use uninitialized object in OnConfigNewType :)
How can I replicate it?
Bug repro
using Ceras;
using System;
using System.Linq;
using System.Reflection;

namespace ReproCerasBug
{
    class Program
    {
        static void Main(string[] args)
        {
            var config = new SerializerConfig();
            config.Warnings.ExceptionOnStructWithAutoProperties = false;
            config.OnConfigNewType = (t) =>
            {
                t.TargetMembers = TargetMember.AllPublic;

                if (typeof(Exception).IsAssignableFrom(t.Type))
                {
                    t.TargetMembers = TargetMember.All;
                }
            };

            //config.ConfigType<MethodModel>().ConstructBy(() => new MethodModel(null, null, null, null));
            //OR
            config.ConfigType<MethodModel>().ConstructBy(typeof(MethodModel).GetConstructors().OrderByDescending(t=>t.GetParameters().Length).First());

            var ceras = new CerasSerializer(config);

            var methodModel = new MethodModel(typeof(string).GetMethods().First(), new[] { typeof(string) });

            var data = ceras.Serialize(methodModel);

            ;
        }
    }

    public class MethodModel
    {
        public MethodModel(Type declaringType, string methodName, Type[] parameterTypes, Type[] genericArguments)
        {
            DeclaringType = declaringType;
            MethodName = methodName;
            ParameterTypes = parameterTypes;
            GenericArguments = genericArguments;
        }

        public MethodModel(MethodInfo method, Type[] genericArguments) :
            this(
                method.DeclaringType,
                method.Name,
                method.GetParameters().Select(t => t.ParameterType).ToArray(),
                genericArguments)
        {
        }

        public Type DeclaringType { get; }

        public string MethodName { get; }

        public Type[] ParameterTypes { get; }

        public Type[] GenericArguments { get; }
    }

}

Using TypeConstruction.ByUninitialized, MethodModel not deserialized again 😃 , i think because setters in not public :)

You'll have to enable serialization of compiler generated fields, and also explicitly target private fields:
config.Advanced.ReadonlyFieldHandling = ReadonlyFieldHandling.ForcedOverwrite;
config.Advanced.SkipCompilerGeneratedFields = false;
config.ConfigType<MethodModel>().ConstructByUninitialized();

and in your OnConfigNewType
t.TargetMembers = TargetMember.PrivateFields;


Chage in serialization output with trailing newline in License field

In a project using
[project]
license = { file = 'LICENSE' }

with the content of LICENSE being
some license

including the trailing newline, with pyproject-metadata 0.8 I obtain a RFC822 serialization of the metadata that looks like:
License: some license\n

with pyproject-metadata 0.9.0b7 I obtain:
License: some license\n
         \n

where I made the newlines explicit in the RFC822 serialization. When the metadata is read back by packaging.metadata.parse_email() the latter results in a license field having content some license\n         . We can argue that whitespace at the end of the text field is not significant, but maybe we should keep stripping the trailing newline.

Also, would it maybe be better to reduce the indentation of subsequent lines in multi-line fields to just one or two spaces? It would made for an easier to read serialization format. And packging.metadata.Metadata() and friends do not strip leading white space from continuation lines, thus having less indentation would make the values obtained from them slightly less awkwards to  look at.

If the license file had a trailing newline, having it in the metadata seems correct. You have to have the trailing whitespace in order to preserve the newline. The email format doesn't actually support newlines - a new line followed by at least one white space is supposed to be merged into one space. The fact tools are keeping them is a bit of a hack.
As for the indentation, I followed setuptools, which indents based on the width of the header key. That would look a little odd coming out of packging.metadata.Metadata. I've also seen 7 spaces, due to the old body mechanism, which was seven white spaces followed by a | in order to preserve newlines.

I think I didn't account for  this part of PEP 639:

If the specified license file is present in the source tree, build tools SHOULD use it to fill the License-File field in the core metadata, and MUST include the specified file as if it were specified in a license-file field. If the file does not exist at the specified path, tools MUST raise an informative error as previously specified.

Which part is not handled exactly? When project.license in pyproject.toml is a string, there should be no License field in the metadata emitted, just License-Expression. Any file matched by the globs in project.license-files goes into License-File fields. I think all this is covered. It is responsibility of the build backend to add the license files into the wheel.

The trailing newline thing is only annoying having to verify the metadata emitted by different versions of pyproject-metadata. But it is not hard to accommodate for the output of different versions. I just wanted to mention it.
Given that (hopefully) multi-line values in metadata fields are going away, I think the exact formatting of the continuation lines is not that important. However, I would like to use a bit less spaces.

Which part is not handled exactly?

If a user has:
[project]
name = "example"
version = "0.1.0"
license.file = "LICENSE.txt"

then Metadata 2.4-supporting backends are supposed to show a warning and fill Licence-File: LICENSE.txt. But it doesn't say if License: should be filled with the contents or left blank.

License cannot exist in metadata version 2.4 and License-File is metadata version 2.4 only. Thus I think Licnese cannot be filled if License-File is emitted with metadata version 2.4.
However, I don't like this feature of the PEP at all. It means that packages get updated to metadata 2.4 only for being built with a different version of the build backend. I think that what is implemented now in pyproject-metadata is much more sensible.
Another thing I don't like in the PEP is that IIUC project.license-files can be used without project.license. In this case, it is not clear to me how an user can easily identify the license, other than comparing the content of the license file with a database of license texts.

It means that packages get updated to metadata 2.4

The PEP doesn't control whether a package decides to use 2.4 or not. AFAICT pyproject-metadata's implementation is a valid application of PEP 639, even if it's not suggested or recommended (or mentioned...).

Another thing I don't like in the PEP is that IIUC project.license-files can be used without project.license. In this case, it is not clear to me how a user can easily identify the license, other than comparing the content of the license file with a database of license texts.

Ah, yes, don't think I noticed that. Not that it's any worse than the current situation, which is a "could mean anything" License field, optional classifiers (which are the only canonical place for license information), and files stuck anywhere in the .dist-info folder. :)

License cannot exist in metadata version 2.4

Ah, if that's the case, that's the answer I was looking for. :)

Though is it?

The legacy unstructured-text License Core Metadata field is deprecated and replaced by the new License-Expression field. The fields are mutually exclusive. Tools which generate Core Metadata MUST NOT create both these fields. Tools which read Core Metadata, when dealing with both these fields present at the same time, MUST read the value of License-Expression and MUST disregard the value of the License field.

That says License and License-Expression are mutually exclusive. And that License is deprecated. It doesn't say License and License-File are, though. And because this is talking about METADATA 2.4, and it describes what should happen (warning on License being set to a SPDX expression), that sort of indicates you can have License in METADATA 2.4.

(Unless I hear otherwise, though, I'm okay with not including it - the only way to hit this in pyproject-metadata is if a backend explicitly sets the metadata to 2.4)

It means that packages get updated to metadata 2.4

The PEP doesn't control whether a package decides to use 2.4 or not. AFAICT pyproject-metadata's implementation is a valid application of PEP 639, even if it's not suggested or recommended (or mentioned...).

What I meant is that a package using meson-python now that uses this license declaration
[prject]
license.file = 'LICENSE'

with pyproject-metadata 0.8 gets metadata version 2.1. However, if the letter of the PEP is implemented, because of the License-File filed that needs to be included, it would get metadata 2.4, which currently cannot be uploaded to PyPI. This would be quite a regression for which no obvious solution exists: new meson-python releases could force-set the metadata version, or depend on pyproject-metadata =~ 0.8, till PyPI accepts metadata 2.4, but there is not way to retroactively apply these fixes to existing meson-python releases.

Not that it's any worse than the current situation, which is a "could mean anything" License field, optional classifiers (which are the only canonical place for license information), and files stuck anywhere in the .dist-info folder. :)

Sure, PEP 639 is better than the current mess, but I had hope for a bit more care in defining the new standard. This lack of attention to detail is the reason why the metadata format is such a mess.

owever, if the letter of the PEP is implemented,
I took it to mean that the PEP only discusses metadata version 2.4. Anything in the PEP is only for packages targeting 2.4. At least I think that was the intent...

reason why the metadata format is such a mess.

This lack of attention to detail is the reason why the metadata format is such a mess. :)

Though is it?

I thought it was, but apparently is not: I haven't read the whole PEP again, but from your quote it seems that it is perfectly ok to produce metadata 2.4 with a License field as long as there is no License-Expression field with it. However, the PEP also says that tools reading the metadata (including package indexes, I believe) should treat License-Expression as higher priority than License not that they should fail validation if both are present. Fun!

I took it to mean that the PEP only discusses metadata version 2.4. Anything in the PEP is only for packages targeting 2.4. At least I think that was the intent...

It would make sense for packages targeting metadata 2.4 to use a proper pyproject.toml structure, wouldn't it? I don't see why someone would require metadata 2.4 but keep their pyproject.toml using the fields defined for metadata 2.1. I would just emit an error if someone requires metadata 2.4 bus uses metadata 2.1 fileds in pyproject.toml.
I think that your implementation is the correct one.

That's assuming that the METADATA 2.4 choice happens in the pyproject.toml, which it normally doesn't. 2.4 happens to be really perfect for auto-selecting the metadata_version, since it changed the pyproject.toml fields. But normally a backend picks this; for example, hatching simply sets METADATA 2.3. So you have to describe what to do when 2.4 is set and the old pyproject.toml values are used. This will also come up with METADATA 2.5+.

On the actual topic of this issue (sorry for hijacking it!), on Discord several people thought the license file should be preserved exactly if specified, including the trailing newline.

That's assuming that the METADATA 2.4 choice happens in the pyproject.toml

Independently that where it happens, I think it should not depend on the version of the build backend (or on the version of the packages on which the build backed depends). That would mean that the build backend cannot be updated to a version supporting the new metadata version untill the package index is not updated to accept it.

On the actual topic of this issue (sorry for hijacking it!), on Discord several people thought the license file should be preserved exactly if specified, including the trailing newline.

I don't mind either way, but I don't see how white space is at the same important (we need to preserve trailing newlines) and not important (leading white space is not removed from continuation lines when metadata is read).

I may have overstated the support, two comments, one in favor of keeping the newline, and one in favor of always adding one. :)
The JSON interface returns the exact original contents.
The Message interface returns something with fixed whitespace after every newline except the first one. You can use inspect.cleandoc to get the oringinal file back... Except it strips all trailing newlines! 🤦. In theory, though, you could figure out and restore the original file exactly, assuming you have a email.message implementation like the one in Python that keeps these. It's also a valid implementation to just join them all, which of course makes this all one line anyway. But then again, this doesn't follow any standard really anyway, with unicode, etc...
I'll check and see what setuptools and hatching do, and if either of those strip, it's probably fine too.
I have no idea why someone thought license.file should fill in this metadata field which has stated (since METADATA 1.1!):

Text indicating the license covering the package where the license is not a selection from the “License” Trove classifiers.

This exactly matches setuptools:
uvx cookiecutter gh:scientific-python/cookie
# Answer questions
cd package
uv build --sdist

Metadata-Version: 2.1
Name: package
Version: 0.1.0
Summary: A great package.
Author-email: My Name<EMAIL_ADDRESS>License: BSD 3-Clause License

        Copyright (c) 2024, My Name.
        All rights reserved.

        Redistribution and use in source and binary forms, with or without
        modification, are permitted provided that the following conditions are met:

        * Redistributions of source code must retain the above copyright notice, this
          list of conditions and the following disclaimer.

        * Redistributions in binary form must reproduce the above copyright notice,
          this list of conditions and the following disclaimer in the documentation
          and/or other materials provided with the distribution.

        * Neither the name of the vector package developers nor the names of its
          contributors may be used to endorse or promote products derived from
          this software without specific prior written permission.

Project-URL: Homepage, https://github.com/org/package
...

But Hatchling strips it:
Metadata-Version: 2.3
Name: package
Version: 0.1.0
Summary: A great package.
Project-URL: Homepage, https://github.com/org/package
Project-URL: Bug Tracker, https://github.com/org/package/issues
Project-URL: Discussions, https://github.com/org/package/discussions
Project-URL: Changelog, https://github.com/org/package/releases
Author-email: My Name<EMAIL_ADDRESS>License: BSD 3-Clause License

        Copyright (c) 2024, My Name.
        All rights reserved.

        Redistribution and use in source and binary forms, with or without
        modification, are permitted provided that the following conditions are met:

        * Redistributions of source code must retain the above copyright notice, this
          list of conditions and the following disclaimer.

        * Redistributions in binary form must reproduce the above copyright notice,
          this list of conditions and the following disclaimer in the documentation
          and/or other materials provided with the distribution.

        * Neither the name of the vector package developers nor the names of its
          contributors may be used to endorse or promote products derived from
          this software without specific prior written permission.

License-File: LICENSE
Classifier: Development Status :: 1 - Planning

(Hatchling also uses License-File and /licenses, and has supported these features for many months from an early version of the PEP.)

FWIW, I like the way the setuptools one looks a bit better with the extra line at the end.

FWIW, I like the way the setuptools one looks a bit better with the extra line at the end.

If we are in "taste for how the output looks like" territory 😄 I don't understand the choice to prefix the lines following the first with a number of spacer equal to the key length plus one, instead than accounting for the column and the whit space and add two spaces. Namely, I find that
License: Foo
         Bar
         Baz

is much more pleasant to look at than
License: Foo
        Bar
        Baz

It really looks like and off-by-one error to me. But it may just be me. 😄

This matches setuptools, I was just trying not to introduce more ways to do things. I agree that an extra space looks better. :) To be fair, I'd also be fine with two spaces, though, if we aren't avoiding adding a new style. @zanieb, did you (or someone else from Astral) say you were working on metadata and you followed the conventions here? Happy to take input if you have any. :)

I like that the current version exactly reproduces in the json versions, so let's go with the current version unless someone feels strongly otherwise.


chore: adding scripts to update owners on L2

Adding scripts to update owners on L2;
Adding to the static audit to make sure the contracts are set up correctly.

lgtm so far


Switch from monarch to TextMate syntax coloring

Otherwise, the semantic highlighting does not work. The monarch styles overrule the inlineClassName used by the EditorDecoration in the monaco editor.

Closed via #138.


wip: Docs/migrate v2

Overview
close #400

PR Checklist

[x] I did below actions if need

I read the Contributing Guide
I added documents and tests.

@manudeli I've kept the page form simple, if you have any feedback or things you'd like to add, please comment to me!

Cool! I'd like to merge this first and update it frequently before releasing v2
If you ok, merge this please

Okay, I'll keep updated after merging


Render image embedded inside a link

A typical clickable build status icon is formatted as below on GitHub (edit the issue to see the raw markdown text):

Right now flexmark only renders the outer link, with inner image not rendered. Any plan to support this?

@robinshine, I use the library for my plugin Markdown Navigator and it renders it, as expected.
Just ran it as a test case and it works. Must be something with settings. Can you share more of how you are setting up the parser/renderer?
[![Build Status](https://ci.tensorflow.org/buildStatus/icon?job=tensorflow-master-cpu)](https://ci.tensorflow.org/job/tensorflow-master-cpu)
.

.
Document[0, 142]
  Paragraph[0, 142]
    Link[0, 140] textOpen:[0, 1, "["] text:[1, 86, "![Build Status](https://ci.tensorflow.org/buildStatus/icon?job=tensorflow-master-cpu)"] textClose:[86, 87, "]"] linkOpen:[87, 88, "("] url:[88, 139, "https://ci.tensorflow.org/job/tensorflow-master-cpu"] pageRef:[88, 139, "https://ci.tensorflow.org/job/tensorflow-master-cpu"] linkClose:[139, 140, ")"]
      Image[1, 86] textOpen:[1, 3, "!["] text:[3, 15, "Build Status"] textClose:[15, 16, "]"] linkOpen:[16, 17, "("] url:[17, 85, "https://ci.tensorflow.org/buildStatus/icon?job=tensorflow-master-cpu"] pageRef:[17, 85, "https://ci.tensorflow.org/buildStatus/icon?job=tensorflow-master-cpu"] linkClose:[85, 86, ")"]
        Text[3, 15] chars:[3, 15, "Build … tatus"]

@vsch Thanks for looking into this.
I am using below code:
List<Extension> extensions = new ArrayList<>();
extensions.add(new UrlResolveExtension());

MutableDataHolder options = new MutableDataSet()
		.set(UrlResolveExtension.BASE_URL, "https://www.google.com")
		.set(Parser.EXTENSIONS, extensions);

Parser parser = Parser.builder(options).build();

HtmlRenderer htmlRenderer = HtmlRenderer.builder(options).build();

String markdown = "[![Build Status](https://ci.tensorflow.org/buildStatus/icon?job=tensorflow-master-cpu)](https://ci.tensorflow.org/job/tensorflow-master-cpu)";
Node document = parser.parse(markdown);
System.out.println(htmlRenderer.render(document));

It turns out that if BASE_URL option is specified, it will not work.

@robinshine, thank you for the detail I was missing. I'll take a look at. If it's a bug I'll make a release with the fix.

@robinshine, did not realize last night that you are using a custom link resolver, so I cannot test it with your setup but suspect that the issue is not in the renderer but the resolver.
If you can give me more information I could possibly be of more help.

@vsch sorry it is my fault. Yes, once my customized url resolver is removed, it renders correctly.
I am using a custom url resolver to render relative urls. And my custom url resolver is as below:
public class UrlResolveExtension implements HtmlRenderer.HtmlRendererExtension {

    public static final DataKey<String> BASE_URL = new DataKey<>("BASE_URL", (String)null);

    public void rendererOptions(MutableDataHolder options) {
    }

    @Override
    public void extend(HtmlRenderer.Builder rendererBuilder, String rendererType) {
    	rendererBuilder.nodeRendererFactory(new NodeRendererFactory() {

			@Override
			public NodeRenderer create(DataHolder options) {
	            return new NodeRenderer() {

					@Override
					public Set<NodeRenderingHandler<?>> getNodeRenderingHandlers() {
				        Set<NodeRenderingHandler<?>> set = new HashSet<NodeRenderingHandler<?>>();
						String baseUrl = options.get(BASE_URL);
						if (baseUrl != null) {
					        set.add(new NodeRenderingHandler<Image>(Image.class, new CustomNodeRenderer<Image>() {

					            @Override
					            public void render(Image image, NodeRendererContext context, HtmlWriter html) {
					            	String url = image.getUrl().toString();
					                ResolvedLink resolvedLink = context.resolveLink(LinkType.IMAGE, url, null);
					                html.attr("src", resolveUrl(baseUrl, url));
					                html.attr("alt", image.getText());
					                html.withAttr(resolvedLink);
					                html.tagVoid("img");
					            }

					        }));
					        set.add(new NodeRenderingHandler<Link>(Link.class, new CustomNodeRenderer<Link>() {

					            @Override
					            public void render(Link link, NodeRendererContext context, HtmlWriter html) {
					            	String url = link.getUrl().toString();
					                ResolvedLink resolvedLink = context.resolveLink(LinkType.LINK, url, null);
					                html.attr("href", resolveUrl(baseUrl, url));
					                html.withAttr(resolvedLink);
					                html.tag("a").text(link.getText()).closeTag("a");
					            }

					        }));
						}

				        return set;
					}

	            };
			}

		});
    }

    private String resolveUrl(String baseUrl, String urlToResolve) {
    	if (urlToResolve.contains(":") || urlToResolve.startsWith("/") || urlToResolve.startsWith("#")) {
    		return urlToResolve;
    	} else {
        	return PathUtils.normalize(PathUtils.resolveSibling(baseUrl, urlToResolve));
    	}
    }

}

I guess the problem exists in  html.tag("a").text(link.getText()).closeTag("a");. Would you please help to point out how to make it working with this scenario?
Thanks!

@robinshine, you don't have a custom link resolver. What you have is a custom node renderer and one that is implemented wrong.
Why not take a look at how the CoreNodeRenderer.java implements its rendering so you can a working start? CoreNodeRenderer
Another place to look is the custom link resolver sample so you can implement a resolver without affecting the rendering?
PegdownCustomLinkResolverOptions
You can safely ignore pegdown options since this only sets the options and extension based on some flags. Beyond that it is all the same code.

Thanks for pointing out the direction. I will check the code. Thanks again and you may close the issue now.


Allow selection of multiple lints

Currently only one lint is allowed at a time, but we might want to enable several lints at a time. Interacts with #5 for non-interactivity and #4 for putting the lint information in the JSON output.

All lints are now used by defaul


fix(logging): flush logs when user identifier changes

Issue #

Description
Ensure that logs are flushed when user signs in or signs out
General Checklist

[ ] ~Added new tests to cover change, if needed~
[x] Build succeeds with all target using Swift Package Manager
[x] All unit tests pass
[x] All integration tests pass
[x] Security oriented best practices and standards are followed (e.g. using input sanitization, principle of least privilege, etc)
[ ] ~Documentation update for the change if required~
[x] PR title conforms to conventional commit style
[ ] ~New or updated tests include Given When Then inline code documentation and are named accordingly testThing_condition_expectation()~
[ ] ~If breaking change, documentation/changelog update with migration instructions~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Can we have a unit test for this change, if possible?


support infinite depth

When converting JSON to PON it currently fails with depths that are greater than 2.
This fix allows infinite depth.

Is this one still needed?


fix #12188 integration-cli: use "top" instead of "sleep" to avoid timeout

Updated most of the sleep tests (there's still a few left where sleep is applicable).
Fix #12188
Signed-off-by: Todd Whiteman<EMAIL_ADDRESS>

@twhiteman It is better to squash your commits. You can do it with git rebase -i

Overall changes are very good.

+1 for squashing, otherwise LGTM


Implement PoC for unaligned cp without persistent channel

What is the purpose of the change
Implements a PoC for unaligned checkpoint without relying on persistent channel.
Brief change log

Checkpoint trigger

The requirement is to trigger checkpoint when network receives the first barrier for some channel. Considering implementation, the netty thread would insert a checkpoint letter into mailbox while receiving the barrier. There are two considerations.

One is how to avoid checkpoint triggered multiple times by different netty threads. We need a global coordinator like CheckpointBarrierHandler to manage it or make every netty thread triggered separately and provide the deduplication judging when mailbox action is really executed.

Another concern is that the task might already read this barrier from one channel before the checkpoint is really executed in mailbox action. So when the task reads the first barrier from one channel, it also needs to execute checkpoint directly as did before, the only difference is without alignment. But we still need to deduplicate the checkpoint when mailbox action is triggered as mentioned above.

Note that we make the assumption that there is only one checkpoint under going every time to make the implementation simple. That means only after one checkpoint is completed, the CheckpointCoordinator would trigger the next checkpoint id.

Insert the barrier in front of partition queue when performing checkpoint.

Note that this should be done before spilling output in-flight buffers, because during inserting into the partition queue, we can meanwhile increase reference with buffers which are spanned over by the barrier and would be spilled later.

How to spill the partition in-flight buffers.

The second step already maintained the referenced collections of in-flight buffers, then we can provide a series of interface methods to fetch them from StreamTask. Also note that the BufferPersister implementation for output side is a bit different with input side. We can get all the in-flight output buffers at a once and add them into persister in batch.

How to spill the input in-flight buffers.

It is quite difficult compared with output side. There are three concerns:

How to spill the unconsumed buffer in deserializer. We provide a interface method for getting it. In practice there are three cases for storing the unconsumed buffer in deserializer: in internal nonSpanningWrapper , SpanningWrapper or spilling to disk. We skip the third case of large record which might exceed THRESHOLD_FOR_SPILLING for spilling , because it is hard to get it now.

The in-flight buffers on input side are dynamic compared with output side, because some buffers are still not visible in channels. We provide a series of interface methods for getting them from RemoteInputChannel. Actually the above unconsumed buffer should be spilled close with the following in-flight buffers in RemoteInputChannel , otherwise they can not be read correctly during recovery, although we do not need to consider it in this PoC.

How to spill the invisible in-flight buffers in RemoteInputChannel. In other words, when the RemoteInputChannel receives a buffer from network, it should decide whether this buffer needs to be spilled by BufferPersister. We need to consider two conditions here. The buffer needs to be spilled if and only if the barrier is not received for this channel and the in-flight buffers in this channel were already fetched by triggered checkpoint before. Note that it is also difficult to guarantee that the buffers belong to one InputChannel would be spilled close with each other.

When to finish checkpoint.

We need to also consider whether all the input and output in-flight buffers are spilled successfully, so we provide the future from BufferPersister to tell the StreamTask when the spilling is finished. The difficulty here is that we need to make the BufferPersister know that when all the in-flight input buffers are already arrived, so we also need a global coordinator to maintain the tags that which channel already received the barrier, and notify the BufferPersister when all the barriers are received.

How to work for LocalInputChannel.

Fortunately we do not need to do much extra works for local channel. We can still rely on the previous way that when the task reads the first barrier from subpartition, it would perform checkpoint directly and the in-flight input buffers are empty in this case.
Verifying this change
Not verified.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

Some explanations for above implementations:

When RemoteInputChannel receives the barrier from network, it would notify BufferOrEventListener which corresponds to UnalignedCheckpointBarrierHandler implementation.

UnalignedCheckpointBarrierHandler would maintain the states of received barrier for all the channels. After one of the channels receives the barrier, it would tag the checkpoint trigger flag as true. When it is notified of receiving a network buffer, then it would check whether the respective channel state is receiving barrier or not before. If have not received barrier for this channel and already triggered checkpoint before, then it would add this buffer to the persister queue.

While mailbox processing input, it would always firstly check the flag from UnalignedCheckpointBarrierHandler. If the checkpoint triggered is set true, then it would execute the real checkpoint process.

In the process of checkpoint done by StreamTask, it would reference all the existing input and output buffers to the persister queue, and then broadcast the barrier as before. It also considers the unconsumed buffer in RecordDeserializer to be persisted.

In the broadcast barrier process, we extend the interface method to indicate whether the added buffer is barrier or not in ResultPartitionWriter. If it is barrier, then we insert it in the head of the subpartition queue.

Some missing parts to be implemented future:

We only define the RecordDeserializer#getRemainingBuffer to interact with process, but have not implemented it in specific SpillingAdaptiveSpanningRecordDeserializer.

We only consider the case of RemoteInputChannel to receive barrier from network to trigger checkpoint. For the case of LocalInputChannel there should be a different way to trigger, and I guess it should not be much complicated.

The current InputPersisterImpl and OutputPersisterImpl are not working well for this PoC because I just copied the previous implementation. It would be a bit trouble for the case of InputPersister because we should guarantee that the buffers within one channel should be written together. Now we only have one global queue in the writer thread of persister and the buffers would be continously inserted into queue after checkpoint triggered.

The async checkpoint logic should consider the state of persister. That means the async checkpoint would finish only after all the input/output buffers are successfully persisted.

We trigger the checkpoint via mailbox calling processInput in tricky way. If we consider the non-blocking output, it would not work because the mailbox would not execute process input if no available buffers. So it is better to enqueue the checkpoint letter to mailbox directly instead of current way future.

We would add some unit tests for covering the changes.

Thanks for the review and concerns @pnowojski !
I updated the codes for addressing most of your comments, and also pasted my previous thoughts from slack channel on the PR description for better understanding the implementation.

What do you mean by:

Insert the barrier in front of partition queue when performing checkpoint.
Note that this should be done before spilling output in-flight buffers, because during inserting into the partition queue, we can meanwhile increase reference with buffers which are spanned over by the barrier and would be spilled later.

?
Also I didn't get this part:

The in-flight buffers on input side are dynamic compared with output side, because some buffers are still not visible in channels.

The "not visible" buffers you mean the buffers that haven't yet been enqueued in the RemoteInputChannel (still in netty/on the wires/not yet sent?).

How to spill the invisible in-flight buffers in RemoteInputChannel. In other words, when the RemoteInputChannel receives a buffer from network, it should decide whether this buffer needs to be spilled by BufferPersister. We need to consider two conditions here. The buffer needs to be spilled if and only if the barrier is not received for this channel and the in-flight buffers in this channel were already fetched by triggered checkpoint before. Note that it is also difficult to guarantee that the buffers belong to one InputChannel would be spilled close with each other.

Do you mean that the only problem is the order of spilled buffers, for correct recovery? I do not see why the decision whether to spill an incoming buffer is tricky. CheckpointBarrierUnaligner is single threaded and has/can have all the knowledge required to make such decision, right?

Again, CheckpointBarrierUnaligner should be able to handle this as well?

So LocalInpuChannel would be spilled only on the output?

What do you mean by:
Insert the barrier in front of partition queue when performing checkpoint.
Note that this should be done before spilling output in-flight buffers, because during inserting into the partition queue, we can meanwhile increase reference with buffers which are spanned over by the barrier and would be spilled later.

I want to emphasis the sequence between broadcast barrier and prepare spill in-flight buffers. In other words, during broadcast barrier to insert it in front of partition queue, we already enter the synchronized lock to easily get which buffers are spanned over by barrier and then reference them into a in-flight collection beforehand. Then when the following operation to get in-flight partition buffers, we can directly get the precise collection even though some buffers are already polled to transport in network. In contrast, if we reference the in-flight partition buffer before broadcasting barrier, it is hard to control that the already referenced buffer must not be transported via network until the barrier is inserted in front.

Also I didn't get this part:
The in-flight buffers on input side are dynamic compared with output side, because some buffers are still not visible in channels.
The "not visible" buffers you mean the buffers that haven't yet been enqueued in the RemoteInputChannel (still in netty/on the wires/not yet sent?).

Exactly. When we received the first barrier from network, the checkpoint is triggered, so for other channels some in-flight buffers might still on the way to RemoteInputChannel. In contrast, we can easily get all the in-flight outputs while performing checkpoint.

How to spill the invisible in-flight buffers in RemoteInputChannel. In other words, when the RemoteInputChannel receives a buffer from network, it should decide whether this buffer needs to be spilled by BufferPersister. We need to consider two conditions here. The buffer needs to be spilled if and only if the barrier is not received for this channel and the in-flight buffers in this channel were already fetched by triggered checkpoint before. Note that it is also difficult to guarantee that the buffers belong to one InputChannel would be spilled close with each other.
Do you mean that the only problem is the order of spilled buffers, for correct recovery? I do not see why the decision whether to spill an incoming buffer is tricky. CheckpointBarrierUnaligner is single threaded and has/can have all the knowledge required to make such decision, right?

Yes, the order of spilled buffers is a big problem atm which is only related to correct recovery. I have some thoughts of it, and we can further confirm it when we want to focus on recovery process. It is indeed a bit tricky for determining whether an incoming buffer is spilled or not, because it relies on another information which can not be provided by CheckpointBarrierUnaligner. The CheckpointBarrierUnaligner only triggers the checkpoint in sync or async way, but it does not know whether the checkpoint was already happened while getting the in-flight buffer from one RemoteInputChannel. When the checkpoint already happened before, the new incoming buffer without barrier has to add into persister queue while inserting into channel queue meanwhile. In contrast, we can only add the incoming buffer into the channel queue and all the existing buffers in channel queue would be fetched during getting in-flight buffers later. So we added another var triggeredCheckpointId  in RemoteInputChannel to indicate this information.
Maybe there are better solutions for integrating the informations inside CheckpointBarrierUnaligner, but i think it is also hard to implement. The current way can make it work although seems ugly.

Again, CheckpointBarrierUnaligner should be able to handle this as well?

Yes, it is indeed handled by CheckpointBarrierUnaligner in current codes. We maintain the numBarriersReceived to decide when to finish the BufferPersister.

So LocalInpuChannel would be spilled only on the output?

Exactly, there are no queued in-flight buffers on local input side.

@AHeise Thanks for the review and I updated the codes to address your comments.

I want to emphasis the sequence between broadcast barrier and prepare spill in-flight buffers. In other words, during broadcast barrier to insert it in front of partition queue, we already enter the synchronized lock to easily get which buffers are spanned over by barrier and then reference them into a in-flight collection beforehand. Then when the following operation to get in-flight partition buffers, we can directly get the precise collection even though some buffers are already polled to transport in network.

Do you mean that we have to already copy references (and bump reference count) of the in flight output buffers, when we are enqueuing the checkpoint barrier (overtaking the buffers). Some of those buffers can be sent overt the network before we actually manage start persisting them?
I guess we can discuss this later in more detail. Maybe an alternate approach would be to atomically enqueue the buffers and at the same time get the collection of in-flight buffers?

Actually the above unconsumed buffer should be spilled close with the following in-flight buffers in RemoteInputChannel , otherwise they can not be read correctly during recovery, although we do not need to consider it in this PoC.

Ok, in that case... :

Actually the above unconsumed buffer should be spilled close with the following in-flight buffers in RemoteInputChannel , otherwise they can not be read correctly during recovery, although we do not need to consider it in this PoC.

Why is this an issue? The order of buffers shouldn't matter that much I think, except of changes in the parallelism. We should always be able to either lazily read the buffers one by one, basically mimicking/reproducing the order how w received them over the network, or we can unspill all of the buffers eagerly, where the order matters even less. Or am I missing something?
If we change the parallelism, we can always read the spilled data twice by different Tasks.

Do you mean that we have to already copy references (and bump reference count) of the in flight output buffers, when we are enqueuing the checkpoint barrier (overtaking the buffers). Some of those buffers can be sent overt the network before we actually manage start persisting them?

Yes, that is the way I did before.

I guess we can discuss this later in more detail. Maybe an alternate approach would be to atomically enqueue the buffers and at the same time get the collection of in-flight buffers?

The current way is also enqueue the buffers and get the collection of in-flight buffers at the same time automatically. What's the difference with your suggestion?

Why is this an issue? The order of buffers shouldn't matter that much I think, except of changes in the parallelism. We should always be able to either lazily read the buffers one by one, basically mimicking/reproducing the order how w received them over the network, or we can unspill all of the buffers eagerly, where the order matters even less. Or am I missing something?
If we change the parallelism, we can always read the spilled data twice by different Tasks.

The key point here is that we might mis-order the buffers from different input channels. In that way we can not deserialize them correctly during unspill because one record might span multiple buffers. But we can tag the channel index as the header for every spilled buffer, then we can concatenate the buffers from the same channel index together during unspill.

The current implementation also enqueues the buffers and get the collection of in-flight buffers at the same time automatically. What's the difference with your suggestion?

Technically not much of a difference. But instead void PipelinedSubpartition#add(BufferConsumer bufferConsumer, boolean finish, boolean insertAsHead), to have Collection<Buffer> PipelinedSubpartition#addInFrontAndGetInFlight(BufferConsumer bufferConsumer)

But we can also spill the channel index as the header for every spilled buffer, then we can concatenate the buffers from the same channel index together during unspill.

Yes, in a way, during the unspilling we could be doing the same thing as current network stack is already doing - segregate the incoming buffers per channels.
Also keep in mind that we are already keeping one record deserialiser per one input channel to handle this exact issue (org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput#recordDeserializers field)

Yes, i agree that the way Collection<Buffer> PipelinedSubpartition#addInFrontAndGetInFlight(BufferConsumer bufferConsumer) seems more elegant, but it needs more refactoring works actually in practice. The broadcast barrier is executed in StreamTask component, from StreamTask to ResultSubpartition it would go through many other components to adapt this new way.
My current ugly way can bypass these components to only work on ResultPartition level. I think we can take your formal way in future.

Also keep in mind that we are already keeping one record deserialiser per one input channel to handle this exact issue (org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput#recordDeserializers field)

Yes, and we already make use of this to guarantee that the spilled buffers in one input channel are close together by StreamTaskNetworkInput#prepareSnapshot. The mis-order issue only exists for the new incoming buffers after performing checkpoint ATM.

FYI: I appended a separate commit for supplementing the ITCase for unaligned checkpoint.


Persist data on docker compose

Problem
When I spin down a docker compose container, the data does not persist.
Solution
When I run docker-compose down and docker-compose up -d I should still have the same data from the previous run.
Describe alternatives you've considered
I have considered persisting a docker volume for the two containers involved, one for Postgres, and the other for the papercups container. However, I am not sure what the minimum directories I need to store are for the papercups container. For example, for Postgres, I can add something like this:
    volumes:
      - ./data:/var/lib/postgresql/data

If I wanted to use a local volume to have the data saved that will need to be mapped over to persist between a down and up command.

Hi @johncadengo I need to do a better job on documentation on the production deploys but I think docker compose prod is a bit misleading. We've just got https://github.com/papercups-io/charts set up and we're gonna recommend you to use this going forward for production. We're going to document the release and upgrade processes soon.

@cheeseblubber thanks for the info. Is there any reason for the move to Kubernetes over docker for production? Does this mean you'll be phasing out support for running this in docker, or that you're limiting it to support for local development, and using Kubernetes for production deployments?


Undefined method `built_in_formatter'

Hello, I've got a strange error using emoji-rspec gem
➜  track2win git:(master) ✗ rspec --require emoji-rspec --format smiles
/Users/anton/.rvm/gems/ruby-2.1.2/gems/emoji-rspec-1.0.1/lib/emoji_test_love/rspec.rb:7:in `alias_method': undefined method `built_in_formatter' for class `RSpec::Core::Configuration' (NameError)
    from /Users/anton/.rvm/gems/ruby-2.1.2/gems/emoji-rspec-1.0.1/lib/emoji_test_love/rspec.rb:7:in `<class:Configuration>'
    from /Users/anton/.rvm/gems/ruby-2.1.2/gems/emoji-rspec-1.0.1/lib/emoji_test_love/rspec.rb:5:in `<top (required)>'
    from /Users/anton/.rvm/gems/ruby-2.1.2/gems/emoji-rspec-1.0.1/lib/emoji-rspec.rb:2:in `require'
    from /Users/anton/.rvm/gems/ruby-2.1.2/gems/emoji-rspec-1.0.1/lib/emoji-rspec.rb:2:in `<top (required)>'
    from /Users/anton/.rvm/gems/ruby-2.1.2/gems/rspec-core-3.0.3/lib/rspec/core/configuration.rb:1024:in `require'
    from /Users/anton/.rvm/gems/ruby-2.1.2/gems/rspec-core-3.0.3/lib/rspec/core/configuration.rb:1024:in `block in requires='
    from /Users/anton/.rvm/gems/ruby-2.1.2/gems/rspec-core-3.0.3/lib/rspec/core/configuration.rb:1024:in `each'
    from /Users/anton/.rvm/gems/ruby-2.1.2/gems/rspec-core-3.0.3/lib/rspec/core/configuration.rb:1024:in `requires='
    from /Users/anton/.rvm/gems/ruby-2.1.2/gems/rspec-core-3.0.3/lib/rspec/core/configuration_options.rb:101:in `block in process_options_into'
    from /Users/anton/.rvm/gems/ruby-2.1.2/gems/rspec-core-3.0.3/lib/rspec/core/configuration_options.rb:100:in `each'
    from /Users/anton/.rvm/gems/ruby-2.1.2/gems/rspec-core-3.0.3/lib/rspec/core/configuration_options.rb:100:in `process_options_into'
    from /Users/anton/.rvm/gems/ruby-2.1.2/gems/rspec-core-3.0.3/lib/rspec/core/configuration_options.rb:22:in `configure'
    from /Users/anton/.rvm/gems/ruby-2.1.2/gems/rspec-core-3.0.3/lib/rspec/core/runner.rb:96:in `setup'
    from /Users/anton/.rvm/gems/ruby-2.1.2/gems/rspec-core-3.0.3/lib/rspec/core/runner.rb:85:in `run'
    from /Users/anton/.rvm/gems/ruby-2.1.2/gems/rspec-core-3.0.3/lib/rspec/core/runner.rb:70:in `run'
    from /Users/anton/.rvm/gems/ruby-2.1.2/gems/rspec-core-3.0.3/lib/rspec/core/runner.rb:38:in `invoke'
    from /Users/anton/.rvm/gems/ruby-2.1.2/gems/rspec-core-3.0.3/exe/rspec:4:in `<top (required)>'
    from ./bin/rspec:16:in `load'
    from ./bin/rspec:16:in `<main>'

bummer... I was hoping to use this too +1

I downgraded to gem "rspec", "~> 2" but I get the same error. What version of RSpec does this work with?

Is this issue repaired? I noticed it was just closed?

+1 on adding a note about RSpec 3 compatibility.


Automatically add padding fields?

First of all great crate, it makes working with wgpu much easier. A question I have is generating unaligned struct fields
struct QuadInstanceData {
    points: array<vec2<f32>, 4>,
    color: vec3<f32>,
}

generates
...
pub struct QuadInstanceData {
    pub points: [[f32; 2]; 4],
    pub color: [f32; 3],
}
const _: () = assert!(
    std::mem::size_of:: < QuadInstanceData > () == 48,
    "size of QuadInstanceData does not match WGSL"
);
const _: () = assert!(
    memoffset::offset_of!(QuadInstanceData, points) == 0,
    "offset of QuadInstanceData.points does not match WGSL"
);
const _: () = assert!(
    memoffset::offset_of!(QuadInstanceData, color) == 32,
    "offset of QuadInstanceData.color does not match WGSL"
);

which fails with
const _: () = assert!(
   |  _______________^
18 | |     std::mem::size_of:: < QuadInstanceData > () == 48,
19 | |     "size of QuadInstanceData does not match WGSL"
20 | | );
   | |_^ the evaluated program panicked at 'size of QuadInstanceData does not match WGSL', test-wgpu/quad_shader.rs:17:15

Isn't it possible to automatically add the padding fields to the generated code instead such that it is always correct? And it is upto the user to either use MaybeUnit or zero out the padding fields. In this case
pub struct QuadInstanceData {
    pub points: [[f32; 2]; 4],
    pub color: [f32; 3],
    pub _pad: f32,
}

would fix the issue

While I could automatically add the padding field on the Rust side, I feel adding fields not part of the WGSL definition that are never read would be confusing. Using types like vec2 or vec3 instead of vec4 doesn't save any space due to alignment requirements. You can either add padding fields to the WGSL struct, use vec4 for the WGSL fields, or use the encase derive instead of bytemuck to perform the padding at runtime.
WGSL defines the memory layout requirements for types that are shared between the CPU and GPU for storage and uniform buffers. wgsl_to_wgpu only requires Rust structs to match the layout in WGSL when using bytemuck. Vertex input structs don't generate those safety checks since the memory layout is defined by the vertex attributes and the requirements are different. I'll try and add an explanation to the readme, since this isn't obvious.

Hmm, I understand. Perhaps the feature can be enabled through another WriteOptions field. And padding field initialisation can be hidden behind a constructor 😄
The way I am working around is adding to wgsl as well so the build generates the padding field in rust as well. But sometimes I'll have to work out the offsets required myself.

Vectors and scalars typically have alignment equal to their size. The only exception is vec3, which has the same alignment as vec4. As long as you stick to vec4 instead of vec3, you shouldn't have too many alignment issues. You can also use encase instead of bytemuck if you still want to use types like f32 or vec3 without worrying about padding.
For your specific example, I would just use vec4 since it's technically the same size after padding.
struct QuadInstanceData {
    points: array<vec2<f32>, 4>,
    color: vec4<f32>,
}

You could even do everything with vec4 and unpack in the shader.
struct QuadInstanceData {
    point01: vec4<f32>,
    point23: vec4<f32>,
    color: vec4<f32>,
}

https://crates.io/crates/encase
https://www.w3.org/TR/WGSL/#alignment-and-size

Correct. There would be no assertions in that case.

I know this is closed, but in case you'd decide to revisit this later, I've added a working draft, to ensure it adds padding fields which makes all the const assertions pass.


how to turn this into http api?

You can see http api in file: effibot/http/http.api
Now, the main branch provide two API:

get /api/answer
get /api/types

You can see them in API file.


Network adapters can send and receive arbitrary message types

Previously I added an AuthMessage type to automerge-repo:
https://github.com/automerge/automerge-repo/blob/830ffb6e57dbdd3c195bbd34fc1392165a44e027/packages/automerge-repo/src/network/messages.ts#L90-L102
We've redesigned the auth provider so that automerge-repo doesn't need to know it exists, so this doesn't belong here any more. But the auth provider still needs to be able to use the network adapter to send its own messages. So the network adapters need to be a bit more agnostic as to the type of messages they'll receive. Not completely agnostic -- we don't want messages to be of type any -- but they should take anything that looks like a message.
This PR replaces the previous Message type, which was RepoMessage | AuthMessage, with the following:
export type Message = {
  type: string

  /** The peer ID of the sender of this message */
  senderId: PeerId

  /** The peer ID of the recipient of this message */
  targetId: PeerId

  data?: Uint8Array

  documentId?: DocumentId
}```

Current dependencies on/for this PR:

main

PR #243

PR #244

PR #242

PR #248

PR #249

PR #250   👈

This stack of pull requests is managed by Graphite.

Current dependencies on/for this PR:

main

PR #243

PR #244

PR #242

PR #248

PR #249

PR #250   👈

This stack of pull requests is managed by Graphite.

Rebased and merged.


Update README.md

Added install webdriver_manager before install selene

Are you sure this is needed?

Selene has webdriver_manager in its dependencies, so it will be installed automatically...
And according to what I remember - it really works like this simple, just by installing Selene without anything else.

@nbdnnm
Hm... now I see that this was the specific case with installing Selene from sources...
But don't you think this problem should be fixed in another way then explicit installation of webdriver_manager?
For example... Selene also depends on selenium webdriver, but seems like there is no problem with it, isn't it?
If yes, then at least we would also add the corresponding line to pre-install webdriver too, not just webdriver_manager...

yep, agree, it should be smarter
don't know how to make it better, will convert it into an issue

Good good! Let's dig deeper into it!
On Wed, Jul 8, 2020, 18:22 nbdnnm<EMAIL_ADDRESS>wrote:

yep, agree, it should be smarter
don't know how to make it better, will convert it into an issue
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/yashaka/selene/pull/220#issuecomment-655587178, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAO6ZHQKHD6QK3XLBJS46LLR2SFLTANCNFSM4OTHRTEA
.


Update distribute-offline-apps.md

https://github.com/MicrosoftDocs/windows-itpro-docs/issues/4022
Changed instructions

@nenonix @JohanFreelancer9 Copy editing is needed for this PR. Thanks.

Thanks @JohanFreelancer9 for reviewing. @Reeced40 can you please verify the steps again? There is an emphasis on Shop settings but the screen shows just the Shop tab, hence, it's Manage > Settings > Shop > Shopping experience.

CC: @nenonix

@jvsam Done, thank you.

Thank you @Reeced40 .
@nenonix @JohanFreelancer9 final copy review? Or good to go? :-) Thank you!

Hi @TrudyHa this pull request has been copy edited and is ready for your final review and approval. Please let us know if there are outstanding issues to be resolved prior to merge. Thank you very much.
Hi @Dansimp if you can help with the review and merging so we can close the issue, much appreciated! Thank you.

Thank you for the quick action @Dansimp. Much appreciated!


Listener doesn't process parallel handshakes

Your environment.
https://github.com/go-ocf/go-coap/issues/109
What did you do?
Just look at last comment. But it seems that other handshakes waits each other.
What did you expect?
dlts.Listener is able to process parallel handshakes.

The issue is here. Every time we get a new inbound stream we block until the handshake is complete.
I don't know what the API should look like. For now I am going to write something quick out of tree that could work.

https://gist.github.com/Sean-Der/a3a6abbe9b78ced613590312a18dc8a4
@jkralik @boaks @jdbruijn this worked^ you just need to decide how many goroutines you are willing to spawn.
Maybe this should be something that Pion automatically does? I am not sure what is idiomatic.

I've checked in my test setup and also works there. See the readme of the repo I've just referenced this in. Will be testing this in my application code, but should work for the time being at lest. @Sean-Der Thanks so much for the quick analysis and fix.

We should look at how crypto/tls buffers internally so that handshakes don't block either.

I have no idea what is the idiomatic but I'll have a look at crypto/tls to see what they do. I can't imagine that handshakes would be blocking there.

You just need to decide how many goroutines you are willing to spawn..

I'll have a look at how the go routines work exactly and what spawning N of them means in CPU, RAM etc. to determine how much I'd be willing to spawn.
For reference to anyone reading: In most applications this wouldn't be that much of an issue since the handshake over an internet connection goes relatively quickly, so context deadlines can be configured to be very tight (e.g. 100 ms). With embedded devices, the full handshake generally takes 5 seconds, which is much more blocking even if it doesn't hang and 500 devices are trying to make connection at the same time.

I'm probably misunderstanding the problem, but can't we use a buffered channel to get around this?
var conns := make(chan net.Conn, 100)

hub := util.NewHub()

go func(l net.Listener) {
    for {
        conn, err := l.Accept()
        if err != nil {
            .. do something ..
            return
        }
        conns <- conn
    }
}(listener)

for {
    select {
    case c := <-conns:
        hub.Register(conn)
}
hub.Chat()

Oh mm, the Accept is on dtls.Listener, so that triggers the whole handshake machinery. We should be able to Accept() on the underlying listener first and then spin the rest off into a goroutine? I suppose the problem is this: https://github.com/pion/dtls/blob/c6dbd482fa3a0a7655e9dc338a0f15be905cdf35/listener.go#L65? We do this inline, instead of spawning a goroutine there that'll then take care of the handshake?

From my point view, spawning goroutine's below to underlayer of dtls/listener, because we want to have similar API as tls.Listener. and it is more convenient for using. Of course there must be option to limit spawned goroutines.

@daenney yep exactly!
@jkralik agree!

Server in crypto/tls doesn't actually do anything so we need to match this API. Instead the first call to Read or Write should start the handshake.

It looks like we will need to do a v3 and add Handshake
I can't believe I missed this before :( this is a pretty big breaking change, but I don't think it will be difficult to do!

If we were to take option 1, realistically we'd want that option on/by default since the current behaviour is undesirable. But we can't flip that on by default since it changes our behaviour quite a bit and we don't know if folks are relying on it somehow. That would result in us having to do a major anyway to be able to flip the default for that option, at which point I'd rather we break the API and align on crypto/tls than take on the tech debt.

Is there any progress on this? We are accepting DTLS connections of potentially many thousand clients (IoT devices) on low bandwidth connections with high timeouts - which is worst case for this scenario ....

My experience with (too) many coap and/or dtls stacks is, it's either intended and developed to be used for IoT, or you may be faced much more pain, then you expect. That doesn't mean, other implementations are bad, it just means, they are not really supporting IoT.

Hi @niondir
Instead of doing your Handshake or first Read/Write in a goroutine create your dtls/Conn in one. Even when we change the behavior to match crypto/TLS you still have to handle the concurrent handshakes.
Happy to create a JSFiddle showing how to do this. When this issue is closed though I don’t think this library will change dramatically, we are just moving when something blocks.
thanks!

FWIW I'm doing like example below in our DTLS server for IoT devices to create 1000 handshake workers/listeners, also using go-coap. Has been working great for over a year now.
func (l *DTLSListener) acceptLoop() {
 	defer l.wg.Done()
 	for i := 0; i < 1000; i++ {
 		go func() {
 			for {
 				conn, err := l.listener.Accept()
 				select {
 				case l.connCh <- connData{conn: conn, err: err}:
 				case <-l.doneCh:
 					return
 				}
 			}
 		}()
 	}
 }

This is a pretty big issue. From my experience, 1 bad handshaker and a wonky config can pretty much take down the entire server (default 30s handshake timeout, client may have a much shorter one like 5s. Client handshakes build up infinitely.)
I will submit a PR, but I propose:
type Handshaker interface {
  Handshake() (net.Conn, error)
}

type handshaker struct {
  inner net.Conn
  conf *Config
}

func(h handshaker) Handshake() (net.Conn, error) {
  return Server(h.inner, h.conf)
}

func(ln *listener) AcceptHandshaker() (Handshaker, error) {
  inner, err := ln.parent.Accept()
  return handshaker{inner, ln.config}, err
}


Add "assumeNearbyYear" option

Rebase / modification of #1556. My hope is that this can be merged.
If the user opts-in by setting options.assumeNearbyYear = true:

When a user enters a 2 digit year, the code now does a better job of
guessing intent. If the 2 digit year is more than 10 years in the future,
it'll use the past century instead. So in the 2015, a value between 00-25
would assume the century is 2000. For 26-99, it'll assume the user meant
1900 (e.g. 1/31/10 = Jan 31, 2010, 1/31/26 = Jan 31, 1926). People don't
generally enter future dates too far in advance, but it's very common
to enter dates from 50+ years ago.

Would it make more sense for options.assumeNearbyYear to allow "false" to skip the behavior (mimicking the current functionality) or an integer which is the threshold for the assumption. So,
options.assumeNearbyYear = false; // exact value used
options.assumeNearbyYear = 10; // if date is more than 10 year in future, then past century is used
options.assumeNearbyYear = 25; // if date is more than 25 year in future, then past century is used
That might provide more flexibility. We could also make the length a separate option.

Thanks Dan, great suggestion. If folks want it, I think a separate option
makes the most sense.
On Thu, Oct 22, 2015 at 11:57 AM, Dan G. Switzer, II <
<EMAIL_ADDRESS>wrote:

Would it make more sense for options.assumeNearbyYear to allow "false" to
skip the behavior (mimicking the current functionality) or an integer which
is the threshold for the assumption. So,
options.assumeNearbyYear = false; // exact value used
options.assumeNearbyYear = 10; // if date is more than 10 year in future,
then past century is used
options.assumeNearbyYear = 25; // if date is more than 25 year in future,
then past century is used
That might provide more flexibility. We could also make the length a
separate option.
—
Reply to this email directly or view it on GitHub
https://github.com/eternicode/bootstrap-datepicker/pull/1631#issuecomment-150270319
.

--
Adam Becker
(951) 9-BECKER
@AdamJacobBecker

That's actually more thoughtful and customization.
I think one option is fine, having two won't benefit much, since when assumeNearbyYear will be false the second option won't be used, and when true the second or its default will be, perforce, used.

I don't feel too strongly about this, but I'd rather the user not have to know the "magic" number of "10" in order to opt-in to this functionality.
E.g. I prefer assumeNearbyYear: true to assumeNearbyYear: 10.

It could accept assumeNearbyYear: true which will set the default value at 10

Great idea -- I just made the modification.

If I may, don't reformat the code please, it make harder to read the diff.

Didn't notice that. The problem is that there is an .editorconfig file with the wrong values. (https://github.com/eternicode/bootstrap-datepicker/issues/1636)

Thanks @ajb!


CFStringCreateCopy not implement?

error LNK2019: 无法解析的外部符号 __imp__CFStringCreateCopy，该符号在函数xxxxx中被引用
error LNK2019: can not resolve the external symbol __imp__CFStringCreateCopy, the symbol is referenced in the function xxxxx
I can find the declaring in head file 'CFString.h', and the 'Foundation.lib' is added in my project,but got a link error.
which lib file should I add to project? Thanks.

using last winobjc build， CFStringCreateCopy seems ok. thanks.


Adding RInstatClimatic Processes to opencdms-process

@faysal-ishtiaq recently updated the Climatol windrose example in this repository to work with OGC API - Processes (using pygeoapi)
@dannyparsons @lloyddewit @ChrisMarsh82 - we can ask initial questions here in this issue:

Should Python wrapper functions validate the parameter values? Validation will be done in the R functions. Is validation at the Python level also needed?
Should unit tests be written for the Python wrapper functions? Unit tests will be written in the R package. Is unit testing at the Python level also needed?
Is the format for how Python wrapper functions should call and interact with R well defined?
What aspects of the OGC Web Process Service standards do we need to be aware of when writing these wrappers?

Based on my current understanding (but it would be good to hear @faysal-ishtiaq's thoughts on these):

The Python wrapper functions should specify the required arguments with type annotations. Validation metadata/documentation must be available (e.g. informing the user of a valid range of values) but the actually validation can be performed in the R code as long as the outcome of validation can be passed back to Python to report to the user appropriately, see this related discussion.
If RInstatClimatic Python wrapper functions exist in this repository then we should be able to execute corresponding tests here - but I see no reason why the R tests couldn't be called from Python
See the Climatol windrose example and current README (which needs improving). Feel free to ask lots of questions.
Hopefully the updated Climatol example goes a long way to answering this question. We should be able to fully answer this through testing that everything works as a user would expect.

Note:

My current expectation is that generic python code for interacting with any suitable R codebase (and generic capabilities for working with pygeoapi) will likely move out of this repository and into opencdms/pyopencdms
Python wrapper functions that are required by OpenCDMS for specific libraries (like Climatol and R-Instat) that are not generalisable can continue to exist in this repository (opencdms-process)
Only a very small amount of R code is likely to exist in the OpenCDMS repositories, the actual R libraries will be mostly maintained by third-parties (e.g. IDEMSInternational/RInstatClimatic).

Additional questions

Should we be developing a template for the wrapper functions to clarify things such as standardized comment headers, type hints, parameter validation, exception handling, coding standards etc.?
How and where are date-time columns requested/produced?

Will data obtained from the Data API always include a date-time column in the same format with the same name?
Can other date-like columns be requested from the Data API? e.g. year, month,
day of the year? We suggest that the R functions can optionally create them if not provided - as long as a date column is given.

Answers

By "template" I assume you mean a documented example of a typical wrapper function showing the expected approach? We certainly need this.
In addition, the intention is to have base classes that will provide the structure (along with useful docstrings). Below are some relevant comments from a recent discussion in the opencdms-process repository:

I think we're going to end up with generic classes like class Process() and class ImageProcess(Process) in pyopencdms, e.g.  from opencdms.process import ImageProcess.
We may then have specific classes like Windrose(opencdms.process.ImageProcess) located in opencdms_process in a climatol module (from opencdms_process.climatol import Windrose).

Yes, the Data API will not provide separate/customisable Y, M and day columns - but these could still be available to processes through "process chaining" where the output of the "date_components" process is then passed to the next process.

@isedwards
Regarding your first two question and current thoughts, I agree with you 100%. As long as, we can see where our tests or validation has gone wrong, we can do it either in Python or R, it really doesn't matter.
Regarding point 3, as of the current code, windrose process will not run because of this line https://github.com/opencdms/opencdms-process/blob/c6ad39f1555869cda44be86708124c1cf450ba2b/opencdms_process/process/climatol/windrose_generator.py#L140, but I kept it intetionally for your comment on this situation. I personally think that the parameters should be well defined.
Regarding question and thought 4, I think we should keep the flexibilty to switch between data source, i.e. csv file or db engine. Regarding R code in this repository, I think any amount of R code is acceptable as long as we don't paste third party R code here and our embedded R code passes all the tests.
I am going through your last comment, please allow me a few minutes.

@isedwards

Additional questions

Should we be developing a template for the wrapper functions to clarify things such as standardized comment headers, type hints, parameter validation, exception handling, coding standards etc.?

I think that we should document as much as we can. When a developer finds a repository with poor/difficult to understand documentation, they will replace it with some other library as long as it is well documented and they don't have to sacrifice much of the flexibility.

How and where are date-time columns requested/produced?

Will data obtained from the Data API always include a date-time column in the same format with the same name?
Can other date-like columns be requested from the Data API? e.g. year, month,
day of the year? We suggest that the R functions can optionally create them if not provided - as long as a date column is given.

I have written some scripts to clean data or to apply expected format to datetime columns. I guess we can start with documenting those.

Answers

By "template" I assume you mean a documented example of a typical wrapper function showing the expected approach? We certainly need this.
In addition, the intention is to have base classes that will provide the structure (along with useful docstrings). Below are some relevant comments from a recent discussion in the opencdms-process repository:

I think we're going to end up with generic classes like class Process() and class ImageProcess(Process) in pyopencdms, e.g.  from opencdms.process import ImageProcess.

I have written a generic dataprocessor class here https://github.com/opencdms/opencdms-process/blob/c6ad39f1555869cda44be86708124c1cf450ba2b/opencdms_process/process/climatol/windrose_generator.py#L94
I haven't put a lot of thought into it, but the way I wrote it, it felt convenient at the moment. Highly appreciate your feedback here.

We may then have specific classes like Windrose(opencdms.process.ImageProcess) located in opencdms_process in a climatol module (from opencdms_process.climatol import Windrose).

Yes, the Data API will not provide separate/customisable Y, M and day columns - but these could still be available to processes through "process chaining" where the output of the "date_components" process is then passed to the next process.

We can document the current datetime format so that people can use it to convert the string to datetime.datetime and then they can manipulate it the way they want.

How and where are date-time columns requested/produced?

Will data obtained from the Data API always include a date-time column in the same format with the same name?
Can other date-like columns be requested from the Data API? e.g. year, month, day of the year? We suggest that the R functions can optionally create them if not provided - as long as a date column is given.

Answers
Yes, the Data API will not provide separate/customisable Y, M and day columns - but these could still be available to processes through "process chaining" where the output of the "date_components" process is then passed to the next process.

@isedwards I assume that the 'yes' refers to the first part of the question and the Data API will always include a date-time column in the same format with the same name.
If we want date-time in a different format for our R processing, then we'll need to convert the data-time info from the Data API format to the format we require.
Is that correct?
Do we already know what the Data API date-time format will look like?
Thanks

@lloyddewit - I believe the API returns ISO 8601 dates. Yes, you will need to convert these to the format that you need
@faysal-ishtiaq - are we close to having a test API end point for Climsoft at api.opencdms.org? (apologies if it's already there and I've forgotten the location - we should improve the service so that it's easier to navigate to the required data)

Apologies if that was the wrong answer... I've been thinking about the web app all day.
If you are making a call from the Python API the data will be a Python datetime object.

@faysal-ishtiaq I just reread your comment above. Is there a reason why the Climsoft date isn't currently returned as a Python datetime object? obsDatetime has DATETIME type in the database. Is it returned as text by SQLAlchemy? If so, could you raise an issue in the pyopencdms repository to discuss this further?

@faysal-ishtiaq - are we close to having a test API end point for Climsoft at api.opencdms.org? (apologies if it's already there and I've forgotten the location - we should improve the service so that it's easier to navigate to the required data)

I implemented it and added some basic tests and it was working. Later Shaibu made some changes and I am not sure about it's current state. I tried to go to https://api.opencdms.org/climsoft but it's returning 404. Probably Shaibu can tell us if there is any modification needed.

@faysal-ishtiaq I just reread your comment above. Is there a reason why the Climsoft date isn't currently returned as a Python datetime object? obsDatetime has DATETIME type in the database. Is it returned as text by SQLAlchemy? If so, could you raise an issue in the pyopencdms repository to discuss this further?

I saw different formats on different tables, so I left them in their respective format. I am raising an issue.

@isedwards The R-Instat R package (now called cdms.products) has a testing infrastructure (https://github.com/IDEMSInternational/cdms.products/tree/main/tests) using the testthat R package, which is now the standard way of implementing tests for an R package.
Note that the default way of structuring an R package means that tests are not included when the package is built and hence not available to call on an installed package without also having the package source code. They are generally seen as a tool for package development as the package build will fail if any tests fail.
However, there is a way to include tests in the package build and I don't see any reason not to do this. So we can do this for cdms.products so that tests can be run via Python by simply calling testthat::test_package("cdms.products") in R.
I'm just raising this point as in general most R packages do not do this and so their unit tests are generally not accessible. While this is only needed for R packages developed specifically for OpenCDMS this won't be a problem as we can ensure that tests are available and avoid duplication of test writing.
Of course it is also possible to write tests externally to an R package with the testthat package and just store them in .R files so we could always write tests for a package if they are not available or the tests do not serve the specific needs. This may mean some duplication of tests doing it this way, so for our own R packages, I think it makes sense to keep the tests internal and make them accessible externally.

@isedwards @dannyparsons
I support @dannyparsons 's suggestion that the Python test calls testthat::test_package("cdms.products") in R.
I understand that this will return a summary of the test results to Python (example here). The Python test could then check the test results using a normal assert.
The R tests would ensure that the R package works as expected. The Python test layer would also need some additional tests to ensure that the data conversions between Python and R are also working as expected. Together these 2 types of tests should fullfill our testing requirements.
The advantage of Python calling the R tests is that it removes test duplication.
In summary, I like @dannyparsons 's suggestion. When he has an (initial) implementation of testthat::test_package, then I'm happy to try and get it working in the Python unit tests.

Thank you @dannyparsons @lloyddewit - this sounds perfect to me.
In general, when OpenCDMS is using an established third-party package, it won't be necessary to run the tests here. However, with this being the first substantial set of processes that are available it is ideal that we're able to demonstrate the full set of working tests from within the opencdms-process repository.


Blank space on right of new terminals

Does this issue occur when all extensions are disabled?: Yes (tested after recording the video)

VS Code Version: 1.91.0
OS Version: macOS 14.5 (23F79)

Steps to Reproduce:

Open a new terminal (editor or panel)
Notice a blank space on the right side of the terminal (seems to scale proportional to window width)
Resize the terminal
Notice the blank space is filled and terminal is sized correctly

Video
https://github.com/microsoft/vscode/assets/6618434/96efea25-672d-41e5-8b88-1a482d5599fe

/duplicate https://github.com/microsoft/vscode/issues/221580


Add appVersion key for kafka

The key "appVersion" is needed for ci testing, it's missing in this yaml. That sometimes will cause testing failure.

/assign @benjigoldberg

/lgtm


[Feature Request]: search with replace

If I don't have overseen anything, there is a search, but no replace feature.
Thank you.

Yes, there is no replace feature, sorry about that. Some kinds of replacements can be done with formulas, or by filtering for values of interest and pasting over them - but there's no general purpose “search-and-replace” feature yet.

Thanks for the hint. Yes, need to start to play with formulas.


Change icon size

How can I change icon size?

@maliatli we implement this functionality in new version of this package animated_notch_bottom_bar: ^1.0.1


Release 0.7.7 throws fatal error "target of assign MUST be an object"

Line: https://github.com/react-native-fellowship/react-native-blur/blob/v0.7.7/src/BlurView.js#L12
Environment:
react-native: 0.13.2,
babel: 5.8.34
Note, it was working fine w/ version 0.7.6
I have another project using react-native 0.17.0 w/ Babel 6, I'll confirm whether or not they same error happens there and update this issue.

Confirmed same error w. react-native 0.17.0 and Babel 6.

+1

+1

It's probably my bad. I need to make a table with RN version to some-rnf-plugins matching. Latest versions relies on babel 6 and rn 0.16+. Sorry for not informing you about it

Oops, missclick

@Kureev  Thanks for looking into it. Just wanted to reiterate that I was able to reproduce the same error with React Native 0.17.0 and Babel 6.

+1

Sorry for the delayed response. Confirmed fixed.