Pin Definitions Conflict with Sketch/Library Variables

I posted this as well on the forum but may be better suited here: https://forum.arduino.cc/t/pin-definitions-conflict-with-sketch-library-variables/1147341
Many libraries (such as Eigen) and sketches may use variables such as A0-A5 and D0-D15 as variables versus pins definitions in pins_arduino.h:

#undef A0
#undef A1
#undef A2
#undef A3
#undef A4
#undef A5
static const uint8_t A0 = PIN_A0;
static const uint8_t A1 = PIN_A1;
static const uint8_t A2 = PIN_A2;
static const uint8_t A3 = PIN_A3;
static const uint8_t A4 = PIN_A4;
static const uint8_t A5 = PIN_A5;

// Digital pins
// -----------
#undef D0
#undef D1
#undef D2
#undef D3
#undef D4
#undef D5
#undef D6
#undef D7
#undef D8
#undef D9
#undef D10
#undef D11
#undef D12
#undef D13
#undef D14
#undef D15
#define D0  (0u)
#define D1  (1u)
#define D2  (2u)
#define D3  (3u)
#define D4  (4u)
#define D5  (5u)
#define D6  (6u)
#define D7  (7u)
#define D8  (8u)
#define D9  (9u)
#define D10 (10u)
#define D11 (11u)
#define D12 (12u)
#define D13 (13u)
#define D14 (14u)
#define D15 (15u)

So if you run a simple sketch:
void setup() {
  Serial.begin(115200);
  while(!Serial && millis() < 5000) {}
  int D0 = 1;
  Serial.println(D0);
  //digitalWrite(D0, HIGH);
}

void loop() {}

it will throw an error:
C:\Users\Merli\AppData\Local\Temp\.arduinoIDE-unsaved2023612-22120-liswj5.0ensn\sketch_jul12a\sketch_jul12a.ino: In function 'void setup()':
C:\Users\Merli\AppData\Local\Temp\.arduinoIDE-unsaved2023612-22120-liswj5.0ensn\sketch_jul12a\sketch_jul12a.ino:4:12: error: lvalue required as left operand of assignment
   int D0 = 1;
            ^
exit status 1
Compilation error: lvalue required as left operand of assignment

A more complex example would be using Eigen:
In file included from
C:\Users\Merli\AppData\Local\Arduino15\packages\arduino\hardware\renesas_uno\1.0.2\cores\arduino/Arduino.h:125:0,
from C:\Users\Merli\AppData\Local\Temp\arduino\sketches\019E3DD99F1ECF6C34FCED459E6CC679\sketch\eigen_example.ino.cpp:1:
d:\Users\Merli\Documents\Arduino\libraries\eigen-main\src/Eigen/src/Core/products/GeneralBlockPanelKernel.h: In member function 'void Eigen::internal::lhs_process_one_packet<nr, LhsProgress, RhsProgress, LhsScalar, RhsScalar, ResScalar, AccPacket, LhsPacket, RhsPacket, ResPacket, GEBPTraits, LinearMapper, DataMapper>::operator()(const DataMapper&, const LhsScalar*, const RhsScalar*, ResScalar, Eigen::Index, Eigen::Index, Eigen::Index, Eigen::Index, Eigen::Index, Eigen::Index, int, Eigen::Index, Eigen::Index, Eigen::Index, Eigen::Index, Eigen::Index)':
C:\Users\Merli\AppData\Local\Arduino15\packages\arduino\hardware\renesas_uno\1.0.2\variants\MINIMA/pins_arduino.h:56:16: error: lvalue required as unary '&' operand
#define D0  (0u)
^
d:\Users\Merli\Documents\Arduino\libraries\eigen-main\src/Eigen/src/Core/products/GeneralBlockPanelKernel.h:1270:73: note: in expansion of macro 'D0'
peeled_kc_onestep(1, blA, blB, traits, &A1, &rhs_panel, &T0, &D0, &D1, &D2, &D3);
^~
........
only one many errors. To correct this pins_arduino.h can be changed to (same way you did it for the analog pins):

// Digital pins
// -----------
#define PIN_D0   (0u)
#define PIN_D1   (1u)
#define PIN_D2   (2u)
#define PIN_D3   (3u)
#define PIN_D4   (4u)
#define PIN_D5   (5u)
#define PIN_D6   (6u)
#define PIN_D7   (7u)
#define PIN_D8   (8u)
#define PIN_D9   (9u)
#define PIN_D10   (10u)
#define PIN_D11   (11u)
#define PIN_D12   (12u)
#define PIN_D13   (13u)
#define PIN_D14   (14u)
#define PIN_D15   (15u)

#undef D0
#undef D1
#undef D2
#undef D3
#undef D4
#undef D5
#undef D6
#undef D7
#undef D8
#undef D9
#undef D10
#undef D11
#undef D12
#undef D13
#undef D14
#undef D15
static const uint8_t D0 = PIN_D0;
static const uint8_t D1 = PIN_D1;
static const uint8_t D2 = PIN_D2;
static const uint8_t D3 = PIN_D3;
static const uint8_t D4 = PIN_D4;
static const uint8_t D5 = PIN_D5;
static const uint8_t D6 = PIN_D6;
static const uint8_t D7 = PIN_D7;
static const uint8_t D8 = PIN_D8;
static const uint8_t D9 = PIN_D9;
static const uint8_t D10 = PIN_D10;
static const uint8_t D11 = PIN_D11;
static const uint8_t D12 = PIN_D12;
static const uint8_t D13 = PIN_D13;
static const uint8_t D14 = PIN_D14;
static const uint8_t D15 = PIN_D15;

this gets rid of the error messages in the same sketch and when using the Eigen Library for instance.  I did test eigen with the recommended changes and it works.
EDIT:  Would have to do for both the Wifi and Minima Version

Many libraries (such as Eigen) and sketches may use variables such as A0-A5 and D0-D15 as variables versus pins definitions in pins_arduino.h:

Hi @mjs513 :coffee: :wave:
How many libraries does that actually affect? Do you know of other libraries? I suggest to keep things in perspective :wink: .
That being said your desired change seems reasonable to me (I'd replace const with constexpr though). Can you please prepare a PR so that we may run it against CI as well as so that other members of @arduino/arduinocore-developers can chime in?

@aentinger
"constexpr" does not work with the R4.  Getting this error when I try using it.  According to what I see its available from c++11 only
pins_arduino.h:75:8: error: unknown type name 'constexpr'

How about you prepare a PR so that we can jointly take a look? :pray: :wink:
For info how to do so, see here.

@aentinger  -  PR Submitted.

Going to close this issue since its taken care of with the PR.


config not working

changing the values in the config/netherite_plus.json doesnt affect values in game.
for example, changing the elytra durabilty to 1000 does not change the netherite elytra's durabilty.
this is the same for all other values.
more info:
this is on a minecraft server.
i am using the most recent fabric version of the mod.

Yes, this is a known thing and is mentioned several times that a restart will be required for certain config values.


Consolidate xunit

Update test projects to reference the same versions of the .NET Framework and xunit.

.NET Framework 4.6
xunit 2.2.0

Codecov Report

Merging #254 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #254   +/-   ##
=======================================
  Coverage   24.31%   24.31%
=======================================
  Files         197      197
  Lines      112379   112379
  Branches    11208    11208
=======================================
  Hits        27330    27330
  Misses      85022    85022
  Partials       27       27

Flag
Coverage Δ

#2015
24.31% <ø> (ø)
:arrow_up:

#2017
24.31% <ø> (ø)
:arrow_up:

#Debug
15.78% <ø> (ø)
:arrow_up:

#Release
14.85% <ø> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c1d0048...39dadf4. Read the comment docs.

@vancem I'm going ahead with the merge on this to simplify some other work. I am not anticipating problems for the following reasons:

This change only affects test code
We already had one test project targeting .NET 4.6, and it wasn't a problem (i.e. no new framework dependency is being introduced)
We already had one test project targeting xunit 2.2.0, and it wasn't a problem (i.e. no new test library dependency is being introduced)


Everything is broken

On linux I'm getting this with every command:
The input text has too few parameters., BadArgCount

The only command that's is triggered is .ai set RQrrOj-UNdEV2_PC5D03US-27MZ7EUtaRH_husjbRQA
but it's not working:
Fetching character info... Error!
 Request failed! (https://beta.character.ai/chat/character/info/)
 Response: Internal Server Error
 Request Content: external_id=1
 Response Content: {"error": "Server Error (500)"}

On windows I get this:
 Error!
 Request failed! (https://beta.character.ai/chat/character/info/)
 Response: Forbidden
 Request Content: external_id=RQrrOj-UNdEV2_PC5D03US-27MZ7EUtaRH_husjbRQA
 Response Content: <!DOCTYPE html>

[...]

I'm using version 1.81

it's not picking up any of my messages with .ai either
.ai hunt is working but it's still not responding to any of my messages, no errors in the console either.

Hello. Please, explain everything you did in steps so I could reproduce it.
Because I can't really encounter any problem

Fetching character info... Error!
 Request failed! (https://beta.character.ai/chat/character/info/)
 Response: Internal Server Error
 Request Content: external_id=1
 Response Content: {"error": "Server Error (500)"}

external_id=1

If you have really specified id with 1, then ofc it would not work.
 Error!
 Request failed! (https://beta.character.ai/chat/character/info/)
 Response: Forbidden
 Request Content: external_id=RQrrOj-UNdEV2_PC5D03US-27MZ7EUtaRH_husjbRQA
 Response Content: <!DOCTYPE html>

[...]

Forbidden

Something like this usually happens when you specify wrong auth token, either servers were simply down for a time

The input text has too few parameters., BadArgCount
On linux I'm getting this with every command:

Please, provie screenshots of actual examples

it's not picking up any of my messages with .ai either
.ai hunt is working but it's still not responding to any of my messages, no errors in the console either.

If you were using auto_setup, checkout this: Additional configuration

Resetting it did the trick. Thank you. As for the BadArgCount, you changed the commands to accept numbers now, that's why it told me that. So !!help and others didn't work. Imo you should just show all 3 pages at once for the help command.
As for setting the char id to 1, i was just testing it out if it would respond at all and got confused with the windows bot I was running.
However, windows still has that error page with auto setup and !!set RQrrOj-UNdEV2_PC5D03US-27MZ7EUtaRH_husjbRQA. I'm using the same config as on linux.

Imo you should just show all 3 pages at once for the help command.

I wanted to, but it's impossible due to text size limit, so I quickly split it with pages but forgot to set page 1 as default. I'll simply make proper modal with buttons later.

However, windows still has that error page with auto setup and !!set RQrrOj-UNdEV2_PC5D03US-27MZ7EUtaRH_husjbRQA. I'm using the same config as on linux.

I can't imagine any reasons for it to act different.
Show me the full Response content of it

01:10:04 Discord     Discord.Net v3.9.0 (API v10)
01:10:04 Gateway     Connecting
01:10:05 Gateway     You're using the GuildPresences intent without listening to the PresenceUpdate event, consider removing the intent from your config.
01:10:05 Gateway     Connected
01:10:05 Gateway     Ready

Starting character setup...
  (Character ID: RQrrOj-UNdEV2_PC5D03US-27MZ7EUtaRH_husjbRQA)
Fetching character info... Error!
 Request failed! (https://beta.character.ai/chat/character/info/)
 Response: Forbidden
 Request Content: external_id=RQrrOj-UNdEV2_PC5D03US-27MZ7EUtaRH_husjbRQA
 Response Content: <!DOCTYPE html>

      <title>Access denied</title>
      <meta http-equiv="X-UA-Compatible" content="IE=Edge" />
      <meta name="robots" content="noindex, nofollow" />
      <meta name="viewport" content="width=device-width,initial-scale=1" />
      <link rel="stylesheet" href="/cdn-cgi/styles/errors.css" media="screen" />

Access denied

            Error code 1020

You do not have access to beta.character.ai.

The site owner may have set restrictions that prevent you from accessing the site.

Error details

            <input id="error-details-checkbox" class="hidden" type="checkbox">

Provide the site owner this information.

               <button class="cf-click-to-copy-btn" id="click-to-copy-btn" title="Click to copy" type="button">

I got an error when visiting beta.character.ai/chat/character/info/.

Error code: 1020

Ray ID: 79cb41cbee14b7b2

Country: ****

Data center: ****

IP: ****

Timestamp: ****

Click to copy

               </button>

        Was this page helpful?
        <button class="border border-solid bg-white cf-button cursor-pointer ml-4 px-4 py-2 rounded" id="feedback-button-yes" type="button">Yes</button>
        <button class="border border-solid bg-white cf-button cursor-pointer ml-4 px-4 py-2 rounded" id="feedback-button-no" type="button">No</button>

        Thank you for your feedback!

               Performance & security by Cloudflare

Some cloudflare bs, as I was expecting.
Gotta try to mess with headers again...

Probably same thing in #19

I have been having this problem with the Windows client as well:
CloudFlare 1020 error.

Well, right now the only solution is to refuse from using it on Windows and switch on Linux/WSL

@FutureQueenBee try to use it with VPN and tell me if problem goes away.
Either, checkout this

When (or if :c) Cloudflare problem will be fixed, this one must go away, 'cause HttpClient is not being used anymore.


ci: add a pull request template

What this PR does / why we need it:
Let's get rid of the checklist that is present in the org's template.
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #
Special notes for your reviewer:
Release note:

NONE

please rebase (if you want i can press rebase, but didnt want without your ack)

/lgtm

/approve


Add support for gzip compression to otel via tonic

Describe the problem you are trying to solve
Add support for configurable compression to OpenTelemetry connectors in tremor.
Describe the solution you'd like
Minimally supports configurable gzip compression and decompression of otel payloads
in conformance with the gRPC over HTTP/2 requirements.
Notes
https://github.com/hyperium/tonic/issues/282
https://github.com/hyperium/tonic/pull/692
Requested by @ekarlso

I believe we would want to provide a new optional field (say "compression") in the configuration for the "otel" connectors.
Note that following the discussion in https://github.com/hyperium/tonic/pull/692 compression is triggered for each and every payload, meaning no per request compression.
Updated Config:
Client
pub(crate) struct Config {
    /// The hostname or IP address for the remote OpenTelemetry collector endpoint
    #[serde(default = "Default::default")]
    pub(crate) url: Url<OtelDefaults>,
    #[serde(default = "default_true")]
    pub(crate) logs: bool,
    /// Enables the trace service
    #[serde(default = "default_true")]
    pub(crate) trace: bool,
    /// Enables the metrics service
    #[serde(default = "default_true")]
    pub(crate) metrics: bool,

    // Do we require `Option` for backwards compatibility?
   pub(crate) compression: Option<CompressionEnum>

}

similarly,
Server
pub(crate) struct Config {
    /// The hostname or IP address for the remote OpenTelemetry collector endpoint
    #[serde(default = "Default::default")]
    pub(crate) url: Url<OtelDefaults>,
    #[serde(default = "default_true")]
    pub(crate) logs: bool,
    /// Enables the trace service
    #[serde(default = "default_true")]
    pub(crate) trace: bool,
    /// Enables the metrics service
    #[serde(default = "default_true")]
    pub(crate) metrics: bool,

   // Do we require `Option` for backwards compatibility?
   pub(crate) compression: Option<CompressionEnum>
}

Compression Enum
pub Enum CompressionEnum {
    gzip,
   // ... add others as requested
}

Heya, this looks good to me :). There is no need for the Option if you implement Default for CompressionEnum and then tag the filed with #[serde(default = "Default::default")] - that way, when missing it's set to the default and the dance around checking if it's set or not is avoided.

Have you evaluated upgrading tonic from 0.6.2 to 0.7.2? If not I would evaluate this, as 0.6.2 seems to only support gzip explicitly with the
<Client>.send_gzip() and <Client.accept_gzip() methods on TonicClent.
While the latest iteration has a slightly different method signature i.e.
<Client.send_compressed(Compression::gzip/..) and Client.accept_compressed(Compression::gzip/) refer here


NextJS headers in JSS 21.5 are not applied

Describe the Bug
We recently upgraded from JSS 21.2.2 to 21.5 and now CORS headers are no longer applied.
I have found the culprit, it seems to be caused by NextJS version 13.4.16, and by downgrading to 13.4.12, the headers work as expected.
To Reproduce
Test a JSS site utilizing Google local fonts in Pages and see CORS related errors on static font files
Expected Behavior
We have added Access-Control-Allow-Origin like this:
/**
 * @param {import('next').NextConfig} nextConfig
 */
const corsHeaderPlugin = (nextConfig = {}) => {
  return Object.assign({}, nextConfig, {
    async headers() {
      return [
        {
          source: '/_next/:path*',
          headers: [
            { key: 'Access-Control-Allow-Origin', value: config.sitecoreApiHost },
            { key: 'Access-Control-Allow-Methods', value: 'GET,OPTIONS' },
          ],
        },
      ];
    },
  });
};

module.exports = corsHeaderPlugin;

With NextJS 21.5, the headers are not added
Possible Fix
A possible fix is to downgrade NextJS 13.4.12
Provide environment information

Sitecore Version: XM Cloud
JSS Version: 21.5
Browser Name and version: Chrome 120
Operating System and version (desktop or mobile): Windows 11
Link to your project (if available):

@larserhardsen Closing this ticket

Fix was provided by a different team by updating Cloudflare which is not directly related to JSS.
The remaining issue is with Vercel which is out of the control of the JSS team and Sitecore.

Just linking to the nextjs ticket for next time I search for this. https://github.com/vercel/next.js/issues/59813


Basic fast filtering implementation

This PR implements the fast filtering algorithm described here.
Fast filtering uses a two pass approach to blur environment maps to create radiance maps. It has incredible speed benefits over traditional importance sampling but comes at the cost of quality in certain scenarios. The implementation in this PR is both faster and higher quality than our current implementation of importance sampling, however, it can only be used to generate a 128x128 radiance map. So the current importance sampling implementation is left in place so users can have crisper environment reflections if they need.
comparison of fast filtering (top, 10ms) and importance sampling (below, 75ms)

I have left the options structure more or less the same. Because there is no longer a low quality importance sampling I removed the "realtime_ggx" project setting and added a setting to choose the quality to use for the fast filtering algorithm. Low quality is default and provides excellent quality for most scenes. High quality can be used for use with especially high detail scenes.
Following this PR I would like to make another one improving the current importance sampling technique so that it is higher quality than the realtime option.

Speed Comparison
Current implementation using Importance Sampling

High Quality
Low Quality

Array
75,000 us*
5,000 us*

Non-Array
20,000 us
670 us

Proposed implementation using Fast Filtering

High Quality
Low Quality

Array
10,000 us*
4,000 us*

Non-Array
2,000 us
300 us

Clang warning:
servers/visual/rasterizer_rd/rasterizer_scene_rd.cpp:829:6: error: logical not is only applied to the left hand side of this comparison [-Werror,-Wlogical-not-parentheses]
        if (!storage->reflection_probe_get_update_mode(rpi->probe) == VS::REFLECTION_PROBE_UPDATE_ONCE) {
            ^                                                      ~~
servers/visual/rasterizer_rd/rasterizer_scene_rd.cpp:829:6: note: add parentheses after the '!' to evaluate the comparison first
        if (!storage->reflection_probe_get_update_mode(rpi->probe) == VS::REFLECTION_PROBE_UPDATE_ONCE) {
            ^
             (                                                                                )
servers/visual/rasterizer_rd/rasterizer_scene_rd.cpp:829:6: note: add parentheses around left hand side expression to silence this warning
        if (!storage->reflection_probe_get_update_mode(rpi->probe) == VS::REFLECTION_PROBE_UPDATE_ONCE) {
            ^
            (                                                     )

@akien-mga Thanks!
Funny enough, that typo would have made the new code run 6 times every frame instead of once and I didn't even notice cause I still had > 60 FPS.

@akien-mga I will push a version with docs later today. Please don't merge until then.

@akien-mga Should be ready to merge pending the outcome of CI.
:)

The build failure looks like an issue with the CI itself.

CI passed but there's a conflict with recent docs changes.

Fixed :)

Thanks!


🤖Sentinel Updates🤖 | 2019-02-27 21:05

Update build definition

Sonar issue reported here: https://github.com/liquigraph/liquigraph/issues/235.


An option to close the menu

Hi @silviomoreto
It will be nice if we have an option to close the menu by $(select).selectPicker('close') $(select).selectPicker('open')
Thanks !

@lovedota for sure that would be great if we have this option. Can you do that and create a pull request?

You can now do this using the toggle method. http://silviomoreto.github.io/bootstrap-select/methods/#selectpickertoggle

You can now do this using the toggle method. http://silviomoreto.github.io/bootstrap-select/methods/#selectpickertoggle

How can we know that dropdown is open or closed ?


Getting issue while loading Phi3 in reft_model

import torch, transformers, pyreft

model_name = 'microsoft/Phi-3-mini-4k-instruct'
model = transformers.AutoModelForCausalLM.from_pretrained(
    model_name, torch_dtype=torch.float16, device_map='cuda',
    cache_dir='./workspace', token='', trust_remote_code=True
)

reft_config = pyreft.ReftConfig(
    representations={
        "layer":15,
        "component":"block_output",
        "low_rank_dimension":4,
        "intervention":pyreft.LoreftIntervention(
            embed_dim=model.config.hidden_size, low_rank_dimension=4
        )
    }
)

reft_model = pyreft.get_reft_model(model, reft_config)

AttributeError                            Traceback (most recent call last)
Cell In[42], line 1
----> 1 reft_model = pyreft.get_reft_model(model, reft_config)
File ~/anaconda3/envs/tune/lib/python3.10/site-packages/pyreft/utils.py:35, in get_reft_model(model, reft_config, set_device)
31 def get_reft_model(model, reft_config, set_device=True):
32     """
33     Create an instance of ReFT model.
34     """
---> 35     reft_model = ReftModel(reft_config, model)
36     if set_device:
37         reft_model.set_device(model.device)
File ~/anaconda3/envs/tune/lib/python3.10/site-packages/pyreft/reft_model.py:14, in ReftModel.init(self, config, model, **kwargs)
13 def init(self, config, model, **kwargs):
---> 14     super().init(config, model, **kwargs)
File ~/anaconda3/envs/tune/lib/python3.10/site-packages/pyvene/models/intervenable_base.py:149, in IntervenableModel.init(self, config, model, **kwargs)
143 if isinstance(
144     intervention,
145     CollectIntervention
146 ):
147     self.return_collect_activations = True
--> 149 module_hook = get_module_hook(
150     model, representation
151 )
152 self.representations[_key] = representation
153 self.interventions[_key] = (intervention, module_hook)
File ~/anaconda3/envs/tune/lib/python3.10/site-packages/pyvene/models/modeling_utils.py:159, in get_module_hook(model, representation)
156     elif representation.component.split(".")[-1] == "output":
157         hook_type = CONST_OUTPUT_HOOK
--> 159 module = getattr_for_torch_module(model, parameter_name)
160 module_hook = getattr(module, hook_type)
162 return module_hook
File ~/anaconda3/envs/tune/lib/python3.10/site-packages/pyvene/models/modeling_utils.py:93, in getattr_for_torch_module(model, parameter_name)
89         current_module = getattr(current_module, param.split("[")[0])[
90             int(param.split("[")[-1].strip("]"))
91         ]
92     else:
---> 93         current_module = getattr(current_module, param)
94 return current_module
File ~/anaconda3/envs/tune/lib/python3.10/site-packages/torch/nn/modules/module.py:1688, in Module.getattr(self, name)
1686     if name in modules:
1687         return modules[name]
-> 1688 raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'")
AttributeError: 'Phi3ForCausalLM' object has no attribute ''

[/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#)` in __getattr__(self, name)
   1686             if name in modules:
   1687                 return modules[name]
-> 1688         raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
   1689
   1690     def __setattr__(self, name: str, value: Union[Tensor, 'Module']) -> None:

AttributeError: 'Phi3ForCausalLM' object has no attribute ''

Can confirm too

@atharvapatiil @kw2828 thanks for bringing up the issue!
the reason is that pyvene does not naturally support phi3 at this point. Please use string access to the intervening module instead. See this as an example:
https://github.com/stanfordnlp/pyreft/blob/main/examples/loreft/train.py#L286
this should allow you to intervene on any torch module.
one downside of this is streamlined ReFT model saving and loading --- since pyvene does not natively support this, you have to come up with your own saving and loading scripts.

for example, try this:
for example, try this:
reft_config = pyreft.ReftConfig(
    representations={
        "component":"model.layers[15].output",   # indicate the component (including the layer) instead
        "low_rank_dimension":4,
        "intervention":pyreft.LoreftIntervention(
            embed_dim=model.config.hidden_size, low_rank_dimension=4
        )
    }
)

First adding this helped
torch.backends.cuda.enable_flash_sdp(False)
torch.backends.cuda.enable_mem_efficient_sdp(False)
and then using this as the string access made it work
wrap the model with rank-1 constant reft
reft_config = ReftConfig(representations={
    "component": "model.layers[15].mlp.output", # string access to the model component
    "intervention": LoreftIntervention(
    embed_dim=model.config.hidden_size, low_rank_dimension=1)})

For more insights you can check out my notebook here https://www.kaggle.com/code/williamalabi/finetuning-gemma-with-reft
Please leave an upvote if you found it hepful:-)

@atharvapatiil @kw2828 thanks for bringing up the issue!
the reason is that pyvene does not naturally support phi3 at this point. Please use string access to the intervening module instead. See this as an example: https://github.com/stanfordnlp/pyreft/blob/main/examples/loreft/train.py#L286
this should allow you to intervene on any torch module.
one downside of this is streamlined ReFT model saving and loading --- since pyvene does not natively support this, you have to come up with your own saving and loading scripts.

@frankaging Thanks for your help, this worked

Thanks for all the inputs! I am closing this issue for now, and feel free to reopen if you have other questions!

I have one question, how to load back phi3 model?
After ReftModel.load :
Traceback (most recent call last):
  File "/gpfs/scratchfs01/site/u/bernerp3/proj/ReFT/eval.py", line 82, in <module>
    reft_model = pyreft.ReftModel.load(
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bernerp3/scratch/conda/envs/dspeed/lib/python3.12/site-packages/pyreft/reft_model.py", line 26, in load
    model = pv.IntervenableModel.load(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bernerp3/scratch/conda/envs/dspeed/lib/python3.12/site-packages/pyvene/models/intervenable_base.py", line 546, in load
    intervenable = IntervenableModel(saving_config, model)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bernerp3/scratch/conda/envs/dspeed/lib/python3.12/site-packages/pyvene/models/intervenable_base.py", line 116, in __init__
    component_dim = get_dimension_by_component(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bernerp3/scratch/conda/envs/dspeed/lib/python3.12/site-packages/pyvene/models/modeling_utils.py", line 100, in get_dimension_by_component
    if component not in type_to_dimension_mapping[model_type]:
                        ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: <class 'transformers.models.phi3.modeling_phi3.Phi3ForCausalLM'>

As I understand in pyvene there is no proper mapping, is it possible to some how solve this?

@bernerprzemek hey! thanks for raising this.
unfortunately, this is a known issue -- pyvene currently does not support to load back model automatically that is not supported natively.
but here are a couple of ways to solve this:

You could take a look at the pyvene tutorial of adding a new model in pyvene without changing the library. Search the keyword: "Add New Model Type".
You could always load back the model manually: you could first construct the original config and create a reft model; and manually load back saved weights by using torch.load or torch.save for saving. All the interventions are saved in reft_model.interventions as key-value pairs.

Thanks a lot of I used second way.
And wow whole ReFT idea is amazing.

for example, try this:
reft_config = pyreft.ReftConfig(
    representations={
        "component":"model.layers[15].output",   # indicate the component (including the layer) instead
        "low_rank_dimension":4,
        "intervention":pyreft.LoreftIntervention(
            embed_dim=model.config.hidden_size, low_rank_dimension=4
        )
    }
)

@frankaging Thank you for sharing an interesting work with us.
I'm also interested in utilizing the module with Phi-3-mini. And I have one question. When you set the config as above, which locations are you specifying? Only the last token of 15th layer?
Thank you in advance!


Basic setup

Hello,
first I'd like to thank you for your work. Pagination is intuitive, documentation is great and available options are well thought out.
I'm backend dev learning how to do frontend and although documentation is just great I cannot figure out how to setup basic project with just one component e.g. Listing which will use uvejs-paginator.
I think that it would be great to add zip file to http://hootlex.github.io/vuejs-paginator/ with package.json, app.js, Listing.vue etc. (minimal setup). This approach would help lots of users :)
I'm using gulp with laravel-elixir and laravel-elixir-vue-2. I've got error with missing vue-html-loader.
One more time thanks for your contribution in Vue world :)

Hello @maciej-bielski and thank you for the kind words.
Have you managed to solve it?

documentation is great

What documentation? There's literally no documentation.


Suggestions in .net7

Here are some suggestions from .net 7. Please take this as they are, suggestions, I'm not saying it's better or worse just some alternatives. Congratulations for this awesome sample!

Use TypedResults and Multiple Results.
In theory Hellang.Middleware.ProblemDetails is not needed anymore since the ProblemDetails are now easy configurable in .net 7
Is SnowFlakIdGenerator since MassTransit already uses NewId with same purposes?
Use required on props of records to simplify/omit some rules.
Question: Should booking service have any reference to a CurrentUser? Will this be called always in a context of a user?

Lately we've been merging the the request, validator, handler in the same class. Since adding or removing the request will almost always involve changing the validator. F12 (Go to definition is magical). We do not use any auto discovery plain old container registration so navigation and find usages work.
public static class Endpoint
{
    public static void MapRoute(IEndpointRouteBuilder route) => route.MapGet("/account/{id:int}", Handle);

    public record Request
    {
        public required int Id { get; init; }
    }

    public class Validator : AbstractValidator<Request>
    {
        public Validator() => RuleFor(x => x.Id).GreaterThan(2);
    }

    public record Response(string Desc);

    public static async Task<Results<Ok<Response>, NotFound>> Handle(
        [AsParameters] Request request,
        ILogger<Request> logger,
        ApiDbContext context,
        CancellationToken cancellationToken)
    {
        if (request.Id == 10)
        {
            return NotFound();
        }

        var entity = await context.Students.FirstOrDefaultAsync(cancellationToken: cancellationToken);

        return Ok(new Response($"everything went fine for: {request.Id}"));
    }
}

In case you want to do unit tests instead of acceptance tests of your endpoints you can inject Queries and Commands instead of db context so you can easily mock.
We've tried TestContainers with server instance per session and each tests gets a new database + migration. It takes 10 sec to start sql server (azure sql edge) and 4ms to create a random new db with 2 tables. Running an acceptance test which: 1. generates new db. 2. creates the tables. 3. starts the api with test server. 4. makes a http get 5. make a query in db. 5. returns response. Takes 1 sec on average. This way ALL tests are insolated and run in parallel (even in the same (implicit) collection).
I'm planning a PoC for TestContainers with RabbitMq where only one instance of rabbitmq is started for a test session (per assembly fixutre) and all tests will use a separate vhost.
We've also dropped MediatR since we have very few truly cross concerns for it's pipeline and the number of request.

OpenTelemetry adds logging to http and mass transit so tracebility is convered by that
Caching at the api level. Since queries/command are never re-used between features they are tailed for each feature need. We add caching explicitly in the handler.
Transactions when need are controlled at the handler level with transient dbcontext. A handler might create multiple transactions or call saves changes multiple times. In case of concurrency issues there are some chances to recover.
Validation happens at the api/endpoint level at least the shallow one anything more complicated is handled directly in the handler

The principles we've been guiding ourselves:

adhere to SOLID principles
everything is an http api. Even cron jobs, background daemons. Use health-checks and probes
vertical slices architecture
invest a lot in identifying the correct bounded context
duplication is far cheaper than wrong/premature abstractization
we do not write applications to have tests, we write tests to build reliable applications. Refactoring should be done no matter how many tests need to be changed. Corollary prefer bdd, acceptance, integration and e2e tests which test business requirements instead of unit tests, especially when the MS are very small. Do sprinkle unit tests when business logic is complex.
observability and traceability through OpenTelemetry are primordial
easy to onboard new developers by self browsing since technical documentation is almost never written/updated.

drive developers into the pit of success with composable extensions framework.
limit magic behaviors: auto mappers, auto registration reflection. Prefer source generators instead.

YAGNI

We will not swap databases without effort
We will not change transport protocols from grpc/http to anything else without effort
We will not switch from an http api to a console app or daemon service without effort.

Hi @narcis-ro, thanks for great suggestion, sorry because of my delay.
For better new managing, this suggestion it was better to create separate issue for each topic.
1- I prefer fail fast compared to functional programming (TypedResults , Multiple Results).
2- I don't familiar with this new feature in .net 7, if it has all feature in Hellang.Middleware.ProblemDetails, it's better we use it with new feature in .net 7. (feel free for creating a pull request)
3- I agree with you, I have plane for using NewId or Ulid in the future. (feel free for creating a pull request)
4- I agree with you, I have plane to change my props to required keywords. (feel free for creating a pull request)
5- We can get some information from token, it can be userId, username or claims in the all microservices.

I will answer in the separate commands for other topics.

1- For endpoint, It is a matter of taste, and each one has its own preference. Your idea is attractive. But my way is common between developer I used this way. But I agree with you and in future I change it to this approach. (feel free for creating a pull request)
2- For test purpose, your approach is parallel, and we can run all test asynchronously, but it's a little slower. But I prefer to reset my database on the collection fixture, because it's a title faster, and it hast to be run sequential.
3- I use mediatr mostly for handling CQRS and use advantage of this approach to improve maintainability and some cross-cutting concern with some pipeline in my code.
4- I don't use open telemetry for logging, in my code, I use serilog for logging http calls. But for handling distributed tracing between microservices, I used open telemetry Instrumentations.
5- For caching, if we need to reuse a cache, it's a good idea to put it inner the handler. But for the simplest use case, my approach is faster and cleaner. (feel free for suggestion in the code)
6- For transaction section, I can't understand what you mean. If you have any suggestion, please send a pull request for clarifying.
7- Validation in endpoint level has a drawback. If we need to call another command, our query in our handler with using your approach, we will skip our validator. But with my approach, validations apply in business level. I use this article REST API Data Validation from kamil.
For principles, I will use your guidance.
Please share to me any good resource by article or code.
Thanks.


added singly list operation code in C lang section

I have added Singly Linked List Operations Code .

@vijay-kumar-yadav do star this repository and follow me for more updates


Management of non-Data Dictionary master metadata: SNOMED CT releases

3++ requirements related to non-DD authored vocabulary .

11/5 - cant do anything with this until we have our own instance of Mauro

21/06/2021 - Not required for 3++

Closed as not required for 3++


Small touches for README.md

Hi,
These are my small suggestions for the text. Let me know what you think.
Dan

I was not good at English so I was saved
Thanks!


WriteOnSerial add delay

I'm trying using your code to interact with Siemens SITRANS gas meter.
I need to enable the RTS/CTS. After doing some test need to add delay on the writeOnserial func.
Adding the delay the meter receive the cmd and respond.
def WriteOnSerial(self, buffer, len):
        txTime = 0.00916666667 * len
        if (self.runningOnRTOS == True):
            #txTime = 0.00916666667 * len
            self._serial.rts = True
            time.sleep(txTime)
            self._serial.write(buffer)
            time.sleep(txTime) # better than flush()! It close RTS immediatly after last sent bit.
        else:
            self._serial.rts = True
            time.sleep(txTime)
            self._serial.write(buffer)
            time.sleep(txTime)
            self._serial.flush()
        self._serial.rts = False

Hi serafimdasaudade, thank you very much.
I'm using an USB-HARTmodem. This modem moves RTS signal by himself.
I send a message to the modem and it provides to set RTS before sending and to clear the RTS after sending the message.
In normal cases, when you need to move RTS, as you have shown, it is better to wait a little while.
Not real time OS are not well syncronized on sending data and RTS management.
Notes.

HART baudrate is 1200. The byte time is more or less 9 milliseconds since each byte is 11 bit long (1startbit + 1stopbit + parity + 8databit).
So "0.00916666667 * len" is the time used to send the whole message.
This is probably too long. Waiting for less time might work.
I don't know if you tried and if you have enough time to try.
I will check as soon as I'll found a modem where I can move RTS by myself.

The modification need to be added only in the "else" since if the OS is real-time, operation should be synchronized.

Before to close the issue I want to try.
In the meantime, thank you again!

Hi serafimdasaudade, thank you very much.
I'm using an USB-HARTmodem. This modem moves RTS signal by himself.
I send a message to the modem and it provides to set RTS before sending and to clear the RTS after sending the message.
In normal cases, when you need to move RTS, as you have shown, it is better to wait a little while.
Not real time OS are not well syncronized on sending data and RTS management.
Notes.

HART baudrate is 1200. The byte time is more or less 9 milliseconds since each byte is 11 bit long (1startbit + 1stopbit + parity + 8databit).
So "0.00916666667 * len" is the time used to send the whole message.
This is probably too long. Waiting for less time might work.
I don't know if you tried and if you have enough time to try.
I will check as soon as I'll found a modem where I can move RTS by myself.

The modification need to be added only in the "else" since if the OS is real-time, operation should be synchronized.

Before to close the issue I want to try.
In the meantime, thank you again!

I'm using a custom modem build in the factory where I work but there are a lot of commercial solution if you search for HART modem in Google. The modem I use has an USB interface and can be used sending HART frames form a PC.
A master HART must works according to the state machine defined by the protocol.
The state machine defines time values in milliseconds. This times are used to switch the state.
A state machine avoids collisions on the network when there are two masters or bursting slaves devices.
On operating systems such as Windows, it is not possible to implement the state machine because Windows is not real-time OS and timing is not accurate.
So in this case, the communication is performed between one master and one slave device without involve the state machine.
For PyHART on Windows a modem with an USB interface is enaugh. USB interface uses a USB/RS232 converter chip. There are latencies and other issues.
In embedded systems, where real-time operating systems are used, the state machine is supported and a modem with RS232 interface can be used.
My idea is to rebuild Linux kernel applying the real time patch and use it on a Raspberry Pi board.
I would like to implement the state machine management in next releases of PyHART.
I want to connect the UART of Raspberry Pi to this adapter or similar (model: ns-rs232-02).

And use the modem Viator MACTek RS232 (Not USB).

[https://www.pepperl-fuchs.com/italy/it/classid_1362.htm?view=productdetails&prodid=89984]
At the end, I want to use the Raspberry board as master and use an external modem connected via UART.
If I understand well, you are not interested in the master communication mechanism but you want to implement a modem on the Arduino board.
You have to modulate in frequency a digital signal. Keep in mind that there are a lot of integrated hart modem chips on the market. I don't know the price.
There are also evaluation boards for these chips.
I don't know which kind of interface are used to send messages to these chips, probably UART.
Regards

I'm using a custom modem build in the factory where I work but there are a lot of commercial solution if you search for HART modem in Google. The modem I use has an USB interface and can be used sending HART frames form a PC.
A master HART must works according to the state machine defined by the protocol.
The state machine defines time values in milliseconds. This times are used to switch the state.
A state machine avoids collisions on the network when there are two masters or bursting slaves devices.
On operating systems such as Windows, it is not possible to implement the state machine because Windows is not real-time OS and timing is not accurate.
So in this case, the communication is performed between one master and one slave device without involve the state machine.
For PyHART on Windows a modem with an USB interface is enaugh. USB interface uses a USB/RS232 converter chip. There are latencies and other issues.
In embedded systems, where real-time operating systems are used, the state machine is supported and a modem with RS232 interface can be used.
My idea is to rebuild Linux kernel applying the real time patch and use it on a Raspberry Pi board.
I would like to implement the state machine management in next releases of PyHART.
I want to connect the UART of Raspberry Pi to this adapter or similar (model: ns-rs232-02).

And use the modem Viator MACTek RS232 (Not USB).

[https://www.pepperl-fuchs.com/italy/it/classid_1362.htm?view=productdetails&prodid=89984]
At the end, I want to use the Raspberry board as master and use an external modem connected via UART.
If I understand well, you are not interested in the master communication mechanism but you want to implement a modem on the Arduino board.
You have to modulate in frequency a digital signal. Keep in mind that there are a lot of integrated hart modem chips on the market. I don't know the price.
There are also evaluation boards for these chips.
I don't know which kind of interface are used to send messages to these chips, probably UART.
Regards


contributor-name-added

contribute here-----
https://github.com/amohata007/Hacktoberfest
and guys do share..

To be on the contribution list you actually have to contribute with something :)
Closing this but you can make a new pull request when you contributed.


Cast params prior to rule execution

Action params should be cast to their proper types as described in the metadata prior to scheduling an action.
This way actions executed from rules can be of various types.

kami : I moved some more code around to reduce duplication and drift. see 5bcdd2728f48b6231d0698077424a339fa7f7b6d

Merging in the change.

:+1:


Added a nice download command in the sidebar

This implements https://github.com/elm-lang/package.elm-lang.org/issues/27
There is a screenshot and some concern whether my little hack is acceptable here: https://github.com/elm-lang/package.elm-lang.org/issues/27#issuecomment-158917680

+1 for the idea.

What is the status of merging this PR? It would be very useful. I often copy-paste the package name from the package title/url.

Probably never, at least not in this form.
I still think it would be a very handy addition to the site, but currently have no time to reimplement it in the current version. Also since I didn't get any feedback last time, I'm not sure whether a contribution in this form is even desired.
This time there would be no excuses for "little hacks" anymore, as elm now can manage focus without JavaScript.


Weird behaviour depending on the capitalisation of executable suffix

On Windows, sometimes the path to an aftman managed executable is returned by tools (such as node-which) with a capitalised suffix .EXE, e.g.: C:\Users\Development\.aftman\bin\stylua.EXE
I'm not sure exactly why its returned like this sometimes, but I've seen it often occur for users, and myself.
When attempting to run this executable directly using the path, there can be odd behaviour.
For example, in cmd:
C:\Users\Development\Downloads\project>C:\Users\Development\.aftman\bin\stylua.EXE --version
stylua 0.18.1

But in powershell:
PS C:\Users\Development\Downloads\project> C:\Users\Development\.aftman\bin\stylua.EXE --version
Aftman error: Tried to run an Aftman-managed version of stylua.EXE, but no aftman.toml files list this tool.
To run stylua.EXE from this directory, add it to one of these files:
- C:\Users\Development\Downloads\project\aftman.toml
- C:\Users\Development\.aftman\aftman.toml

But C:\Users\Development\Downloads\project\aftman.toml contains the contents:
# This file lists tools managed by Aftman, a cross-platform toolchain manager.
# For more information, see https://github.com/LPGhatguy/aftman

# To add a new tool, add an entry to this table.
[tools]
rojo =<EMAIL_ADDRESS>selene =<EMAIL_ADDRESS>wally =<EMAIL_ADDRESS>luau-lsp =<EMAIL_ADDRESS>stylua =<EMAIL_ADDRESS>wally-package-types =<EMAIL_ADDRESS>
I would expect the tool to always run fine, even with .EXE as a suffix.
Looking at the code, the offending issue seems to be current_exe_name() not stripping the .EXE suffix like it does for .exe, meaning aftman doesn't find it in manfiest.tools vector
https://github.com/LPGhatguy/aftman/blob/412d5b5a54971a881bfd99ebd7976b48cf5dedfe/src/main.rs#L73C1-L83C2

That's a funny bug! I wonder where those tools are conjuring a different extension from.
I think a reasonable fix would be to also try to strip the uppercase version of EXE_SUFFIX. I want to avoid introducing new bugs on platforms that tend to have case-sensitive filesystems if we can. What do you think about that solution?

Sorry for the delay in responding!

I wonder where those tools are conjuring a different extension from.

Looks to just be how node-which defines the extension on windows platform https://github.com/npm/node-which/blob/25c5191ec4e29d8f83028fae5f52c0182d006d96/lib/index.js#L33

What do you think about that solution?

Sounds reasonable to me. Maybe even gate it to only windows platforms?


ios8 buttonDidSelected crash

let rotation = Init(CASpringAnimation(keyPath: "transform.rotation")) {
$0.duration        = NSTimeInterval(1.5)
$0.toValue         = (0)
$0.fromValue       = (Float(-180).degrees)
$0.damping         = 10
//$0.initialVelocity = 0
$0.beginTime       = CACurrentMediaTime() + delay
}

iOS is not supported - the docs incorrectly stated this. See https://github.com/Ramotion/circle-menu/issues/27


Clean up code after data orging

moved a bunch of files around so I am using this as an opportunity to clean up code

Done so far

field_chemistry/00_colors_and_groups
field_chemistry/01_chem_prep
field_chemistry/02_chla_graphing


fix docker docs by adding /tmp and net info

closes #748
need to bind mount /tmp and also specify a netowrk to accomplish
anything useful

Pull Request Test Coverage Report for Build 1204

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 71.21%

Totals

Change from base Build 1203:
0.0%

Covered Lines:
11932

Relevant Lines:
16756

💛  - Coveralls

Pull Request Test Coverage Report for Build 1205

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 71.21%

Totals

Change from base Build 1203:
0.0%

Covered Lines:
11932

Relevant Lines:
16756

💛  - Coveralls


Fix: gracefull solution not used (e.g fail use Aurora for FLAC)

E.g fail use Aurora for FLAC if  tag does not support this because solution (e.g Aurora) marked as {desired: false} and {support: false} when {canPlay: true} too.
What i need:
setMedia MP3 or OGA or FLAC
jPlayer params {supplied:"mp3,oga,flac"}
If browser doesn't support format in html5 use aurora as solution
{solution: "aurora", auroraFormats:"flac"}
And here is problem:
if {solution: "html,aurora", supplied:"mp3,oga,flac"}
mp3:  plays
oga:  plays
flac: fail
if {solution: "html,aurora", supplied:"flac"}
mp3:  fail
oga:  fail
flac: plays
if {solution: "aurora", supplied:"mp3,oga,flac"}
mp3:  fail
oga:  fail
flac: plays
P.S
Please, check code around this place, i can't understand it. E.g
$.each(this.solutions, function(solutionPriority, solution) {
    if(solutionPriority === 0) { // Why first solution already desired?
        self[solution].desired = true;
    } else {
        var audioCanPlay = false;
        var videoCanPlay = false;
        $.each(self.formats, function(formatPriority, format) {
            if(self[self.solutions[0]].canPlay[format]) { // Why [0]? Maybe [solutionPriority]?
                if(self.format[format].media === 'video') {
                    videoCanPlay = true;
                } else {
                    audioCanPlay = true;
                }
            }
        });
        self[solution].desired = (self.require.audio && audioCanPlay) || (self.require.video && videoCanPlay);
    }
});

Oops, I just saw this PR at the moment. I wrote a similar one (https://github.com/happyworm/jPlayer/pull/365), not sure which one is the best.

@Phyks, JPlayer is dead, use simple custom wrapper over  tag and Aurora polyfill

@imShara do you have any feedbacks on such solutions? Have you tried Howler.js for instance?
This issue affects Ampache a lot and we are hesitating between trying to fork and kind of patch jPlayer or just move to another solution.

@Phyks, I made myown simple solution with plain js api. Actually, we don't need any frameworks to play sound.


Server / connection errors mid-progress may cause endless error loop

This relates to the issue outlined in https://github.com/flutter/flutter/issues/61546.
When a connection goes away during a upload, the the response sink currently generates almost endless stream of errors.
Instead, the error should be reported only once and further transfer should be cancelled.

This should be fixed in 2.0.0-beta.5 as well.


Add ALP encrypted images

Need to move this to draft, as encrypted image is failing to build and we have inconsistent build numbers.
https://build.opensuse.org/package/live_build_log/SUSE:ALP/ALP:kvm_encrypted/images/x86_64

ALP is dead


Consolidate frontend

Description:
This moves all frontend related static assets to the polymer repo. Building scripts will also be moved to the repo. We will no longer contain a version of the frontend ready to go when doing a git checkout. Frontend will still be included in the PyPi package.
This PR is the first step. Probably some more cleanup PRs will follow after this.
Polymer PR: https://github.com/home-assistant/home-assistant-polymer/pull/468
To do:

[ ] Update Dockerfile to build frontend
[ ] Update Hass.io to build frontend

Will pip3 install -e . used in script/setup still work?
I'm thinking of a developer who just wants to develop backend:
script/setup will make hass point to the git-checked-out-version, but that version now doesn't have FE files.

No. One will no longer be able to work on backend stuff without compiling the frontend once.
To make it easier to compile I'll include a Dockerfile. In a future PR we could add a script that takes the latest release from PyPi and extracts the frontend.

We're not going to merge these PRs before 0.56 release.

Will one be able to work on backend stuff if they don't have frontend: in their config?
I thought the intent of this PR was to decouple FE from the BE?

One can always work on any backend stuff without loading the frontend. You will just need to depend on tests instead of playing with it in the UI

Brainstorming idea:
What if we published the prebuilt polymer files on PyPI as an independent package? It could then be registered in frontend component using the existing REQUIREMENTS system. If the frontend component isn't configured with development: 1 it uses the resources from the PyPI package.
It's obviously messy that we'd have a PyPI package that only contains the polymer code, but we already include the frontend files in our current package. This would just be further separation of the two repos.

Tbh, I would actually prefer to have some way to get the system running without first fighting quite a bit to get the front-end stuff to compile & working. However, it feels wrong to ship some auxiliary binary package to pypi. Therefore I would encourage that we need find a simple way to get UI bits compiled in a simple manner (wrt. pip install x) instead of going for dependency hacks on some pypi snapshot from the frontend.
In my opinion providing an option to use docker for this is not enough. Just my 0.02e, I applaud the idea of this PR completely so do not take this as a disagreement :-)

I'm mainly concerned with raising the barrier to entry for contribution, and personally would prefer some channel to ship the precompiled frontend to developers. If we ballpark by issue/PR count, over 95% of our contributions are to the backend. If people aren't actively developing the frontend, I don't see why we need everyone to rebuild it from source themselves. Especially since we are attempting to isolate the frontend from the backend, it makes sense that we'd move toward shipping it as it's own product.
Docker certainly will make building the frontend easier, but it's a big additional hurdle to people who are often starting their first foray into open source, and are not familiar with all of the tooling. I just want to make sure we carefully consider the ramifications of the final solution on casual developers.

I like the idea of publishing it as a pypi package a lot!

I've added initial work to convert it to a Python package: https://github.com/home-assistant/home-assistant-polymer/pull/468/commits/ef7911cfd5b27563f7b8d54ad3d406f19244921e

@armills last commit has changed it to importing the frontend from a PyPi package. It will not use the PyPi package when you pass in a path to the polymer repo:
frontend:
  development_repo: ../home-assistant-polymer

Will still need to fix the tests.

Looks great!
Now, if I make a FE change and I would like to compile it and check that compiled code works - how do I do that?

Good question 🤔
So one way I have been doing it now is to run python3 setup.py sdist in polymer dir to generate a package. Then with HA virtualenv active, pip3 install ../home-assistant-polymer/dist/home-assistant-frontend-20171021.2.tar.gz --upgrade. Then start HASS with hass --skip-pip
I do realize that that's suboptimal. Guess we need another config option to load the compiled version from the repo. Shouldn't be too hard but for the sake of getting this monster merged, can we do that in another PR?

Yeah this shouldn't block merging this PR
Maybe put a sticky message (not only regarding compiling FE, but about the whole new dev process) in #dev_frontend and #dev chat until this is fixed and new documentation is pushed to the site.

Also, brace yourselves for recommits of the polymer submodule in every PR.

Fixed the tests, this is ready for review.

@armills @andrey-git shall we merge this?

http:
  development: 1

is still used for cache control.
Expected?

When I want to run with FE dev mode I set development_repo in configuration.yaml pointing to home-assistant-polymer checked out from git.
I also run script/bootstrap there. Do I have to run script/build_frontend too?
It seems so, since mdi.html and serviceworker are not in the repo - they are only built by build_frontend.

Yes, build the frontend once. Removing Dev option from http component is on todo


feat: 🎸 add onStatusChange to confidential transaction

Description
allows for subscription to confidential transaction status changes
Breaking Changes
none
JIRA Link
✅ Closes: DA-995
Checklist

[ ] Updated the Readme.md (if required) ?

:tada: This PR is included in version 24.0.0-confidential-assets.1 :tada:
The release is available on:

npm package (@confidential-assets dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Feature add bailingmm

Hi, @ChuanyangZheng ,
Currently, VLMEvalKit only supports the evaluation of public accessible MLLMs. For BailingMM, I failed to find any information or the usage on the Internet.


Producer config doesn't accept "acks" or "request.required.acks"

I'm using Nuget package Confluent.Kafka v0.9.2-preview.
Config setup is essentially:
var config = new Dictionary<string, object>();
config.Add("bootstrap.servers", "[valid brokers here]");
config.Add("acks", 1);  // tried "all", 0, etc.

The exception that is thrown:
  System.InvalidOperationException was unhandled
  HResult=-2146233079
  Message=No such configuration property: "acks"
  Source=Confluent.Kafka
  StackTrace:
       at Confluent.Kafka.Impl.SafeConfigHandle.Set(String name, String value)
       at Confluent.Kafka.Producer.<>c__DisplayClass18_0.<.ctor>b__2(KeyValuePair`2 kvp)
       at System.Collections.Generic.List`1.ForEach(Action`1 action)
       at Confluent.Kafka.Producer..ctor(IEnumerable`1 config, Boolean manualPoll, Boolean disableDeliveryReports)
       at Confluent.Kafka.Producer`2..ctor(IEnumerable`1 config, ISerializer`1 keySerializer, ISerializer`1 valueSerializer, Boolean manualPoll, Boolean disableDeliveryReports)
       at Console.Producer.Program.Main(String[] args) in C:\Projects\tests\Kafka\ConfluentKafka\src\Console.Producer\Program.cs:line 19
  InnerException:

I have tried both "acks" and "request.required.acks" both have the same result.
Looking at the documentation and code - it would appear this should work.
Am I doing something wrong?

FWIW other producer specific config args do work (e.g. 'client.id', 'max.in.flight.requests.per.connection', 'retries', etc).

acks is a topic-level configuration property, so you need to set it on the default.topic.config sub-dict of your configuration dict.
Here's an example (for the consumer, so it has other properties) that shows how to define the sub-dict:
https://github.com/confluentinc/confluent-kafka-dotnet/blob/e41f56cc7892da3382b29c1bca731456f5791d37/examples/AdvancedConsumer/Program.cs#L41

Worked great - thanks!


Fix check percentage exists

Hi, first of all thank you for making great feature to vim-lsp and lightline.
I've got error when open rust file with rust-analyzer.
Error detected while processing function lightline_lsp_progress#progress:
line    8:
E716: Key not present in Dictionary: percentage
E15: Invalid expression: s:lsp_progress['message'] != '' && s:lsp_progress['perc
entage'] != 100
E716: Key not present in Dictionary: percentage
E15: Invalid expression: s:lsp_progress['message'] != '' && s:lsp_progress['perc
entage'] != 100

https://user-images.githubusercontent.com/56591/107960301-cd741f00-6fe7-11eb-9661-b87af0f910b7.mp4
Please check this and include 🙏

Thank you!!


DATA-1982 - Remove AddBinaryDataToDatasetByFilter in API

We're no longer implementing this

@tessavitabile Yep! This is not used anywhere and was unimplemented, so this should not break anything :)


eth: remove methods to retrieve uncle counts

fixes (by avoidance) #114
Removes the following methods related to uncle counts:

eth_getUncleCountByBlockHash
eth_getUncleCountByBlockNumber

Post-merge, methods related to uncles don't make much sense since per EIP-3675 it is not possible for a block to have ommers (uncles).

No, i would keep it for retrieving this information for blocks before the merge. Once 4444 is implemented, only full archive nodes could keep it.

This information is still available via eth_getBlockBy*. Currently we don't even have eth_getUncleBy* methods, so seems a bit moot to have the count getters.

Not super attached to this - just an idea. Just because it is removed from the spec doesn't mean that clients won't continue to support it for a long time. Just no longer a requirement.

Also FWIW the existing uncle data available via RPC isn't enough to actually recreate the block fully from the JSONRPC data, since as you mentioned you need the eth_getUncleBy* methods, and those aren't currently in the spec.
So if anything, I'd advocate for adding in the uncle getters, which work totally fine w/o these count RPCs (you can always use eth_getBlockBy* and simply count the hashes in the uncle fields.
Of course I'm not saying we actually should include the uncle getters, just that I agree 100% these count getters are definitely not worth including.


wss stopped working, cannot figure out the reason

We used Ocelot 18.0 + SignalR on localhost environment, it was Angular 13.x.
We use self-signed certificates and everything worked smoothly with this simple configuration:

We upgraded to Angular 15 (the corresponding version of SignalR NPM has been updated accordingly) - I'm writing this just to emphasize there was no other relevant changes that come on my mind. And it does not work now. I cannot figure out why and probably you would give me some hint?

Could it be somehow related to Ocelot or I should exclude it from suspicion list?

Hi Alexandis!
Thanks for your interest in Ocelot!
A lot of people tries to route wss-protocol traffic via Ocelot gateway, but they failed! 🤣
Here are other issues related to WSS:

#1636
#1662
#1509
#1364
etc.

Are you sure that Ocelot v18 + WSS SignalR setup is working?
Officially Ocelot does not support WSS protocol at all. You will find nothing related to wss-protocol in documentation!
More about: Websockets
I would recommend you to connect to the service directly without Ocelot.
Finally,
Could you let us know that your direct connection to the service is working please?
Any logs, screenshots are welcome!
After that we will continue discussion...

Aha, and one more update!
We need to keep in mind always that if Microsoft has provided new version of .NET, for instance .NET 6 to 7 release, then with high probability we have breaking changes and some new features in the System.Net namespace. Also the System.Net.WebSockets namespace can have new changes. For example the ClientWebSocket class can have significant upgrade to support latest changes of SignalR.
So, Ocelot has no injectable packages to vary WebSockets and SignalR between .NET versions.
This is the major problem of Ocelot for now!

Answering your first question - yes, suprisingly the shown configuration allowed us to befriend wss and Ocelot 18, working on localhost without hitch.
And then - thank you for the last comment: I made a direct connection and ensured the error is the same. Which probably means that "Websockets feature is broken during .NET version upgrade"... :(

I'm glad that my explanations helped you!
Yes, rolling back to Ocelot v18 is good idea to get wss-traffic being routed.
But I'm surprised also, that old Ocelot supports wss proto.
Who knows maybe I will reopen this issue soon, because it has interesting specification Ocelot v18 + SignalR npm package for Angular.
Are you sure that Angular 15 and SignalR NPM for nodejs are compatible? What exact npm are you using?
Finally, the team and me, are interested in support of last release of Ocelot (v19, .NET 7 release).
I'm afraid we can accept bugs for latest version only. Old Ocelot packages/releases are not supported by our team!
Could you upgrade the gateway solution to v19.0.2 release please?
And let us know the user scenario results once again please!

I think I will have to put wss thing on hold, because we have now received a hi-priority tasks.
But I have noticed yesterday, that this piece of code is not triggered anymore (request for wss protocol is not intercepted in NET7?):
               options.Events = new JwtBearerEvents
               {
                   OnMessageReceived = context =>
                   {
                       if (context.HttpContext.Request.Path.StartsWithSegments("/signalr-hubs/notification"))
                       {
                           if (context.Request.Query.TryGetValue("access_token", out StringValues accessToken))
                           {
                               context.Token = accessToken.First(); // this wss request authentification used to work in NET6 for sure...
                           }
                       }
                       return Task.CompletedTask;
                   }
               };

So probably this is the reason of the error I mentioned.

I want to finally clarify what actually happened in my case.
The reason of my error was actually irrelevant to Ocelot. It was related to the overflown JWT token: too much data was packed into it and the token was actually invalid.
But the mechanism with adding token to URL to authenticate wss request really works with Ocelot provided the configuration above. The question of security having token in URL is another matter and needs to be discussed separately. However, there is no other way to authenticate wss request in another way, as far as I know...

Yeah, yeah!
Ocelot can route requests even with invalid tokens in query params! :rofl:
Also, what is the problem to move auth-token to headers or even to body?
If you have an intention to come back to us with well-defined WSS feature request in future, we will accept it with pleasure.

From what I've heard lately, wss does not support passing token via header or body - only URL.

Alexandis,
    options.Events = new JwtBearerEvents
    {
        OnMessageReceived = context =>
        {
            if (context.HttpContext.Request.Path.StartsWithSegments("/signalr-hubs/notification"))
            {
                if (context.Request.Query.TryGetValue("access_token", out StringValues accessToken))
                {
                    context.Token = accessToken.First(); //authenticating WSS request
                }
            }
            return Task.CompletedTask;
        }
    };

I've searched for the source code in Ocelot and I didn't find something similar.
I guess this coding recipe can be useful in Ocelot...
What issue have you resolved by this code?
It looks like you forgot to register authentication, authorization services in these Ocelot features:

Authentication
Authorization
Websockets

Do you think we need to develop Websockets somehow?
My understanding is that in your coding recipe you forward auth-token from query params to be authorized by downstream WSS service, right?
I need just understand is it implemented in Ocelot or not?
Is your user case covered by JWT Tokens authenticationProviderKey and AddJwtBearer method?

@raman-m thank you - we indeed do not use these extensive Ocelot configuration options. Actually, I was not the one in the team implementing Ocelot, so I don't have much detail about it. Probably these configuration options might come in handy for us if we get the issue while publishing our solution in test Azure environment (where we do not have SSL certificate and will need to use that "ignore" thing for accepting self-made certificates)... I do not know in advance actually, sorry.
As to my piece of the code - it is to authorize the wss request with incoming URL token on the site where SignalR hub actually resides on. It is not connected to Ocelot, but probably might be useful for someone if he meets the issues with SignalR functioning...

Aha...
Could you read and review the Websockets chapter please?
After my review I see that Websockets vs SignalR don't support authentication & authorization at all.
Look into this section: Websockets | Not Supported
The points 11, 12 say that authentication and authorization are not supported.
I guess it is true. But I need to dive into the source code deeply to understand current situation...
If points 11, 12 are still actual then it means we can provide some basic authorization support for downstream services via JWT token forwarding like it is implemented in your coding recipe. But for sure this recipe must be generic without any hardcoded options and/or settings.
What do you thing?
Also, could you upload your solution to GitHub please? You can remove real hosts for security, just replace them by localhost.
Pay attention that some real and useful solutions can be added to Ocelot as web app sample to folder: samples
Also sample project/solution can show us the possible approach and design, and we could enhance old feature or develop new feature.


Docker-compose: Oracle container could not start after stop

I see this error after docker stop and start again and oracle did not start (run docker-compose as root user on centos 7)
# docker -v
Docker version 17.06.0-ce, build 02c1d87
# docker-compose -v
docker-compose version 1.15.0, build e12f3b9

# docker-compose stop
# docker-compose up -d
# docker-compose logs -f

found files in /u01/app/oracle/oradata Using them instead of initial database
rm: cannot remove '/u01/app/oracle-product/12.1.0/xe/dbs': Directory not empty
Here is my docker-compose.yml

oracle:
image: sath89/oracle-12c
container_name: oracle
restart: always
environment:
- CHARACTER_SET=TH8TISASCII
ports:
- 1521:1521
volumes:
- ./database:/u01/app/oracle
- /etc/localtime:/etc/localtime:ro

I also found this error on database recreation, Is this normal ?
Database not initialized.

The problem is they are not recommend to use loop-lvm on production.
https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/
Finally, we might need to setup direct-lvm mode or recreate filesystem using -n ftype=1 to solve this problem.

Closing this issue because its not related to this image


Import/Export CSV/JSON

Thank you for this great project :tada:. This is a feature request (hopefully that's welcome):
In order to make migrating from existing systems easier, it would be helpful to be able to upload a CSV (or similar) file in the admin dashboard to create users and records inside a collection to initially populate the database. Thank you!

I like the idea of having "Import/Export to CSV" for a collection (json would be easier, but I think we can make CSV to work too).
Also I think this could be done without too much changes (probably even client-side in the admin UI entirely).
My only concern is how the file field values will be handled (downloading them from absolute urls and then upload them to the PocketBase file store?), but I'll have to think more about it and will consider it for future releases.

I don't have a CSV preference, JSON would work! Files would definitely be challenging, downloading from absolute URLs to the PB file store like you said would work if it's not too complicated. Not sure if it's any easier, but you could force users to manually dump files into the PB file store with specific names that match filenames in their JSON import. The import functionality would be useful even if it doesn't handle files at all initially; that could always be added in a later phase (depending on how you feel about releasing partial features).

Could be handled by the create apii method allowing a JSON list of JSON objects which conform to the table definition. Then you could use anythiing to POST a JSON list to the create method (curl, wget, sdk, whatever). This is how Supabase allows for bulk uploads to tables.

+1 for this 'import schema' feature, It will better having a 'export schema' feature too.
It's very useful for integrate testing or CI job, for now I have to add a db file to my git repo.

This would be great

This turned out a little more complicated than I initially anticipated.
Experimental support for only import/export of collections configuration from JSON (both from the UI and as a automated migation command) was added in v0.4.0 release (and even only this ended up troublesome since we have to handle the referenced collection and schema ids which will defer when created manually on different environments).
During my inital POC development for "Import/Export of data" I stumbled on several issues (file uploads are not even the main problem here):

There is no unambiguous automated way to import records with relation fields (the referencing record ids or even the collections may not even exist).
Allowing to import large data sets from the UI (or while the application is running) could block other db writers, which practically could make the application unresponsive during the import process.
Because cross references between 2+ records are allowed, we cannot import them one by one since it will fail the validations. Bulk importing collections configuration have the same caveat, but usually collections are a lot smaller in size (eg. a couple MB top) and the changes can be quickly applied, which for the records data this may not be the case.

So I'll need to spend a little more time to research other approaches how to handle better the import/export of records data properly.

The requirement for bulk upload, means that multi transactions are needed.
This is also related to the need for relations and doing writes that update many collections.
One way to do this is to support 2PC commit style transaction coordinator.
A byproduct of this is that you can also create many read only mysql with a single write only db, and hence get scale out.
There are a few golang projects that are working on this.
Litestream: https://github.com/benbjohnson/litestream is a sqlite replicator. It stores the mutation is s3 and have backup and restore from s3.
LiteFS: https://github.com/superfly/litefs is also a sqlite replicator that uses Fuse, and sync directly between the mast and slave.
LiteFS is watching for Transactions at the Fuse level and so it possible that is can also do a type of 2PC, in which it watches for transactions that are bounded, and then holds them all, and then commits when given the command to commit.
The docs on this aspect are here: https://github.com/superfly/litefs#lite-transaction-files-ltx

Hi @ganigeorgiev, do you have an ETA for this task?

@catalinux17 No, sorry.
Currently the ongoing users refactoring from #376 is with highest priority because it blocks everything else.
After that we'll continue following the existing roadmap and data import/export from the Admin UI eventually will be handled after implementing transactional API operations within a single or multiple collections.

I wonder if there is any option to import existing SQL schema and data into new pocketbase DB. Without this feature is moving more complex data structures and data to pocketbase too time-consuming. Pocketbase seems to be great when starting new project, but i see in docs no info on importing data from existing DBs. Am I missing something?
I have a project which to me seems to be perfect fit for pocketbase, but with around 500 columns and 100 tables, schema and data import is considerable concern.

@mirozahorak
I don't think there is a generic solution that will fit all cases when dealing with project data migrations because 1-1 mapping is not always possible.
If you use PocketBase as framework, you can write Go migrations that will be embedded in your final executable - https://pocketbase.io/docs/migrations/ (including creating collections programmatically - https://pocketbase.io/docs/collection-methods/#create-new-collection)
Or if you use the prebuilt PocketBase executable, prepare plain SQLite migration scripts and then run them against your db file (pb_data/data.db). Each collection represent an individual table and all fields are translated to db columns.
The best approach if you are planning to migrate your existing project would be to try PocketBase with a clean state locally, create some collections and records and explore the existing db schema. And then based on the internal structure, plan and prepare the dump/migration scripts.

Thanks for the answer.
I have tried to simply modify data.db sqlite database pocketbase created in DBBrowser for SQLite. I added fields to existing table (collection) but they are not showing in admin.
I guess migration is the only option.
It would be great to have a tutorial for this, especially for people who don't use GO and want to use pocketbase with javascript or dart.

The introduction guide allows downloading a prebuilt binary of PocketBase wich is based on this go file https://github.com/pocketbase/pocketbase/blob/master/examples/base/main.go.
When extending PocketBase (aka. using it as framework) you can follow the guides under the "Extend" section - https://pocketbase.io/docs/use-as-framework/.
PocketBase is distributed as a Go package and you have to use it and import it in the the same way as any other go library (aka. import "github.com/pocketbase/pocketbase).
If you are not familiar with Go, you can also check the official A Tour of Go guide to get familiar with the basics of the syntax.
Feel free to open a new Q&A discussion if you need further help with the migration, but once again, my recommendation would be to first play around with a clean PocketBase state in order to explore its features, supported fields and evaluate whether actually it can be used for your use case. Also note that PocketBase is still under active development so backward compatibility is not guaranteed and it may not be suitable if you are don't want to keep up with eventual future breaking changes (although I try to keep them minimal, they are sometimes are unavoidable due to the lack of my insight on the various use cases that different users have).

I'm working on simple import tooling using the JS SDK (sadly I'm not that proficient with Go). Far from perfect, but it should let you import basic CSVs already. I'll be adding support for flat JSON files in the coming days. Just a temporary solution to help you guys get started.
One feature that I'm missing while implementing this is batch insert endpoint, with large datasets and a remote database it's not feasible to create 1 record per request (imagine a 100k+ records table created with 100k+ requests over a network). I hope that it will become a thing for the integrated feature.

@michal-kapala If you want to import records via the terminal you can also make use of the .import command in the sqlite3 cli tool, something like:
sqlite3 pb_data/data.db '.import test.csv foo --csv'

The main blockers in this issue are outlined in https://github.com/pocketbase/pocketbase/issues/48#issuecomment-1211775627.

I hope e2e-testing (using a tool such as Playwright) will be easier once this is implemented. For example I want to be able to delete all records in a collection and re-seed it with testing data using the API (not using migrations).

@michal-kapala If you want to import records via the terminal you can also make use of the .import command in the sqlite3 cli tool, something like:
sqlite3 pb_data/data.db '.import test.csv foo --csv'

The main blockers in this issue are outlined in #48 (comment).

@ganigeorgiev sadly simple table import does not create a collection, or at least it doesn't appear in the dashboard, so for that to work you'd need to modify your data (at least)
I've added JSON support to my pocketbase-import, works pretty decently. I recommend using JSON over CSV as a safer option. A quick look on what the tool does and what it doesn't:
Pros

Honors system columns (id, created, updated)

Column name conflicts resolved with _ prefix

Explicit collection structuring

field types assigned by regex matchers
no constraints or rules added on import

Customizable CSV parser options

Cons

Slowness and network throttling - creates 1 record per request due to API limitations, inserting a 100k rows collection takes a few minutes with local instance
Currently limited PB type support
Currently CSVs with complex text fields may not import properly
id cannot be populated by the user - _id is created instead. If this is a no-go as for your requirements, you will probably have to modify the database with SQLite CLI or tools like DB Browser for SQLite

If anyone is looking for a semi-automatic way to import data from an existing SQLite DB into Pocketbase, I wrote two scripts to do so for my own use case:
export: https://gist.github.com/trevorstarick/664d4a40c0246eced6b178890d073096 import: https://gist.github.com/trevorstarick/31defb0c288735669530c684855d61b1

Nice
would like to extend this to support continuous import using a CDC ( change data capture ) pattern.
My data lane is JSON based and I want to support data streaming into Pocketbase.
The use case is AI / ML Workflows.
But it can also be used for general programming .
It will be reentrant styje so that you as a code can push changes to the data lane and have it arrive in your Pocketbase instance.

Does anyone have a work around for this? I will implement this in a few days
I already have an implementation, https://github.com/shynome/xlsx-importer

@ganigeorgiev  I noticed this issue is still blocked and I would like to offer some suggestions.

For numbers one and three,

There is no unambiguous automated way to import records with relation fields (the referencing record ids or even the collections may not even exist).
Because cross references between 2+ records are allowed, we cannot import them one by one since it will fail the validations. Bulk importing collections configuration has the same caveat, but usually collections are a lot smaller in size (eg. a couple MB top) and the changes can be quickly applied, which for the records data this may not be the case.

I suggest that the importation be done without validations and foreign key checks. I got this idea from Drift, an sqlite library that is used to build flutter apps. In the documentation, the author suggests disabling the foreign keys before you start a migration and enabling them after the migration is complete. He also suggests running a 'PRAGMA foreign_key_check' to ensure that there are no inconsistencies in the database before re-enabling the foreign key checks. Considering that data importation is almost as good as a db migration, I was wondering if you can adopt something similar to this.
In my opinion, if there is no way to validate the data or determine the order in which the tables should be imported (because of weird relationships and dependencies), you could simply put a warning that the inbuilt data importer does not validate the data being imported. The developer could always write a script to validate the data themselves, write a manual migration or use the web APIs if they want their data to be validated as it is imported into pocketbase.

For number two,

Allowing to import large data sets from the UI (or while the application is running) will block other db write operations, which practically could make the application unresponsive during the import process.

Since I doubt anyone who has read the documentation of pocketbase will try to import a large data set while the application is running in production, I think a simple warning listing all these possible issues should be enough. If you don't want the UI to seem unresponsive when doing bulk imports, you can just create a full screen page that shows the progress of the import but does not accept any input. Not accepting any input means the app does not have anything to respond to and therefore cannot be unresponsive. I will also suggest that all web UI links redirect to this page whenever a bulk upload is in progress.

@ganigeorgiev Thank you so much for all your effort on this application. I would really like to hear what you or anyone thinks about my suggestions.

@fbede While nice to have, this feature request remains a very low priority and will be implemented at later stage after the upcoming internal refactoring and the bulk/transactional web APIs support.

@ganigeorgiev There's no problem. Thank you for your work so far on pocketbase.

hope upload with csv, so that we can upload woocomerce products to pocketbase

@fbede While nice to have, this feature request remains a very low priority and will be implemented at later stage after the upcoming internal refactoring and the bulk/transactional web APIs support.

What is the status of that refactor?
also QQ: if I wanted to use an LLM to just generate some JSON schema in order to import a basic structure without relations or data, like just a very basic lift I could then dial in, what meta-schema/standard should I point to or give an example of?
https://pocketbase.io/docs/js-collections/#create-new-collection ?
is the complete enough?

Hi @ganigeorgiev,
Would that be possible to simply add the ability to inject javascript plugin files into the UI admin ? (potentialy add in like in a collection). That's a bit a of an 'I've just thrown this together' feel, but that would be sufficient to add some basic features, based on available rest APIs.
What do you think of it ?

@evaisse I don't see how your request is related to the above discussion. For Admin UI extendability please check https://github.com/pocketbase/pocketbase/issues/898 and https://github.com/pocketbase/pocketbase/discussions/4355.

@evaisse I don't see how your request is related to the above discussion. For Admin UI extendability please check #898 and #4355.

Yes you're right, I was suggesting that we could probably deal with CSV import/export directly in the UI admin just by the addition of 2 buttons and a javascript to pipe the upload of download of data for a given collection.

Perhaps this could help someone. I have been using PocketHost for a while and between my dev and publishing, I developed this tool. This helps one to sync data from a local pockethost and any other other pocketbase.
All the best.

Looking forward to this feature - primarily for the transactional support (all or fail and rollback). Great project and really enjoying using it thus far!

not sure if anyone is still looking for a good solution here, but I'm using DuckDB with the sqlite extension. That gives you very robust CSV handling, you can manipulate with SQL using an in-memory DB and then copy to your pocketbase DB as required. It's a very powerful, very lightweight tool

Great tip will check it out. Thanks.

I have in my todo to eventually add at least a UI option for client-side generated exports for selected records similar to the logs but remains a very low priority for now.

I will try to keep my import tools compatible with the latest release until then, might turn them into a pr one day


initBlueskyComments always returns undefined, even if no bluesky post was found

It'd be really helpful if the initBlueskyComments function could return a boolean or something if it is rendering anything (like the error or loading messages). I'd like to show a placeholder if I haven't made a bluesky post yet for a specific blog post and I need some kind of indication from the function to do that.

@donquxiote I implemented this functionality in the latest release via callback. See the README section here
Can you let me know if that meets your needs?


Tm/fix dockerfile

Motivation
The build were failing because I messed up Dockerfile in #114
Approach
Make sure that Dockerfile is not missing anything

📣 NOTIFICATION
You are receiving this message because we did not publish any packages.
Generated by @thefrontside/actions


Suddenly Homarr container points at hostname:port instead of local IP:port, breaking proxy

Environment
Docker
Version
0.15.7
Describe the problem
I've been using Homarr for about 2 years now as docker container. With the following compose, it automatically pointed at <IP_ADDRESS>:7575, which is the IP of the Orange Pi 3 LTS on which it's running.
Suddenly it started pointing at http://orangepi3-lts:7575, breaking the proxying in Nginx Proxy Manager.
All other services on the same machine point at the IP, Homarr doesn't. What can I do to fix it?
EDIT: To my surprise, after several failed connections, Homarr is now proxied to my tld, despite NPM being set to IP and Homarr using "orangepi3-lts".
Logs
services:
  homarr:
    container_name: homarr
    image: ghcr.io/ajnart/homarr:latest
    network_mode: "host"
    restart: unless-stopped
    volumes:
     - /srv/dev-disk-by-uuid-aeae213f-8ce4-405c-9d96-db90e69c28f8/Config/homarr/configs:/app/data/configs
     - /srv/dev-disk-by-uuid-aeae213f-8ce4-405c-9d96-db90e69c28f8/Config/homarr/icons:/app/public/icons
     - /srv/dev-disk-by-uuid-aeae213f-8ce4-405c-9d96-db90e69c28f8/Config/homarr/data:/data
     - /var/run/docker.sock:/var/run/docker.sock
    ports:
      - '7575:7575'

Context
Exporting hostname...
Migrating database...
yarn run v1.22.19
$ tsx ./migrate.ts
Done in 2.25s.
Starting production server...
Listening on port 7575 url: http://orangepi3-lts:7575
Warning: data for page "/board" is 155 kB which exceeds the threshold of 128 kB, this amount of data can reduce performance.
See more info here: https://nextjs.org/docs/messages/large-page-data
â„¹ Local icons repository directory exists and contains 80 icons

Please tick the boxes

[X] I confirm that I attached the proper logs
[X] I've read the docs
[X] I've checked for duplicate issues
[X] I've tried to debug myself

Did you recently perform an update of homarr?
Otherwise the issue may lie somewhere else.
Maybe it has to do to you using (or lack of use of) the env var "NEXTAUTH_URL"

I let Watchtower take care of updates, so I don't know if it's related.
I do not use NEXTAUTH_URL
BTW as I wrote, it somehow works, seems NPM understands that orangepi3-lts means <IP_ADDRESS> and lets me reach it via Cloudflare tld.

orangepi3-lts most likely is being translated to "localhost" in your host file in some capacity, or docker networking is setting that up.
If Homarr is indeed being proxied, is there still a problem in functionality?
I personally don't really look that much into what's written in that part of the log, my own instance has a random set of characters that mean nothing and change at every restart, yet my proxy manager handles it without issue.

orangepi3-lts is the name I assigned to the machine when I installed Open Media Vault on it, I don't know how OMV manages localhost names.
Anyway no, the problem vanished on its own, this can be closed.
The only thing that worries me a bit is:
Warning: data for page "/board" is 155 kB which exceeds the threshold of 128 kB, this amount of data can reduce performance.
My Homarr is actually quite slow, especially when enabling config mode to add/remove apps. Can I do anything to mitigate that?

Ah, I see, no I don't think I'll multiply boards, the whole point is to have everything in one place.
I'll resist until 1.0.
For what's worth, this is my board

goodness that's a lot of apps.
One thing that slows the page load time a lot is actually the ping function, also to be improved in V1.0.
Out of curiosity, could you turn off pings and try to reload? See if load time diminishes.

Yes, disabling pings made it snappier for sure, I tried Edit mode and clicking the cogwheel on apps made the settings dialog appear immediately. It took 10-15 secs before.
I will try to do what you say with bookmarks, never thought about it, I have indeed links that are set as apps, just for habit.
Thanks for the advice.


Custom inline snapshot matcher will always pass a test

🐛 Bug Report
A custom inline snapshot matcher does not fail when the snapshot mismatches. The tests will pass and leave behind the updated snapshots
To Reproduce
Follow https://jestjs.io/docs/en/expect#custom-snapshot-matchers and add
const {toMatchInlineSnapshot} = require('jest-snapshot');

expect.extend({
  toMatchTrimmedInlineSnapshot(received) {
    return toMatchInlineSnapshot.call(this, received.substring(0, 10));
  },
});

it('stores only 10 characters', () => {
  The snapshot will be added inline like
  expect('extra long string oh my gerd').toMatchTrimmedInlineSnapshot(
    `"extra longe"`
  );
});

Expected behavior
Changing the inline snapshot will lead to a failed test.
Link to repl or repo (highly encouraged)
https://github.com/eps1lon/jest-custom-inline-snapshot-matcher
envinfo
  System:
    OS: Windows 10 10.0.18363
    CPU: (6) x64 Intel(R) Core(TM) i5-9400 CPU @ 2.90GHz
  Binaries:
    Node: 14.10.1 - C:\Program Files\nodejs\node.EXE
    Yarn: 1.22.4 - C:\Program Files (x86)\Yarn\bin\yarn.CMD
    npm: 6.14.8 - C:\Program Files\nodejs\npm.CMD
  npmPackages:
    jest: ^26.4.2 => 26.4.2

This is a documentation issue. The following patch will correctly fail the test:
-toMatchTrimmedInlineSnapshot(received) {
+toMatchTrimmedInlineSnapshot(received, inlineSnapshot) {
-  return toMatchInlineSnapshot.call(this, received.substring(0, 10));
+  return toMatchInlineSnapshot.call(this, received.substring(0, 10), inlineSnapshot);
},

Docs change is proposed over at https://github.com/facebook/jest/pull/10889


[zh-cn] sync architecture/nodes.md

content/zh-cn/docs/concepts/architecture/nodes.md

/approve

/lgtm


Add Indie Game documentary

Add "Indie Game: The movie" on Documentaries.

Hi there. Thank you for your contribution. Unfortunately, this is a duplicate of PR #62. Thanks again.


[Front-Door] Routing rule creation needs improved error messages

I lost a fair bit of time trying and failing to create a simple Redirect routing rule because I kept receiving this generic catchall message :

"usage error: [--redirect-type {Moved,Found,TemporaryRedirect,PermanentRedirect}][--redirect-protocol {HttpOnly,HttpsOnly,MatchRequest}] [--custom-host CUSTOM_HOST] [--custom-path CUSTOM_PATH] [--custom-fragment CUSTOM_FRAGMENT] [--custom-query-string CUSTOM_QUERY_STRING]"

I had to dig into the code to find the below line with those specific rules which were preventing me from creating my rule (I had copy pasted a --backend-pool parameter and it couldn't communicate that that wasn't compatible with a redirect rule).
https://github.com/Azure/azure-cli-extensions/blob/6c38daed3de23239b94131a4268bd92448e1bfb3/src/front-door/azext_front_door/custom.py#L493

Front-Door

A redirect routing rule means that frontdoor would return a 300 level status code directly to client, without forwarding the request to your backend. Thus, it is not possible to specify a backend pool for a redirect routing rule.

@DanoThom I believe you have received answers to your question, so checking in if this issue can be closed. Please let us know.


Improve missing image error message

Improve messaging to match a bit more like what it is like in Lightwave:

Updated reporting to this:

Not exactly the same as lightwave, but it bring you to the next window where one can search for a new directory


New Project :  Minion Translation App

I would like to add a Web Application to your project, It would have the following features:
It would translate the English conversation into minions native language.
It uses pure VanillaJs and takes the help of API to get the translated text.
Please assign me this issue under GSSoC 2021

@signifershrey  Sounds Exciting! Is there any API for this?

@991rajat  Check this one -  api.funtranslations.com

@signifershrey It's not working But Believe you will be able to do it.
Project sounds interesting too.
Shall I assign you this?

Yes, assign me!

@signifershrey Sorry for this there was some confusion Someone developed this project I will merge it today by itself what you could do is add new feature to it or a new language will assign it to. Is it Okay?
I let you know as soon as possible

Okay!

Hey @signifershrey Sorry for the Delay Minion App has been added.
Feel free to ping me the Idea you would Like to work. I'll be Happy to assign you.
And change the title also.

@signifershrey Any update?, Kindly reply within 2 days or I have to close it.

@signifershrey Kindly reply in 24 hours,

Minion Translation Already added. If @signifershrey Want to create a new type or modify existing project Create a new Issue.
We are closing this one.


Fix encoding of xml node namespaces and improve encoding tests.

Fix encoding of keys returned by XML::Node#namespaces. The key parts were encoded like the document, but should be UTF-8 per definition. The value parts were already encoded as UTF-8.
Encoding of the various text returning functions was tested against the document encoding, but it should be UTF-8 per definition. The XML/HTML documents are now changed to a non-UTF-8 versions, to make the tests more meaningful.
This obsoletes #1385  - although the same content, the attached patches are better separated and so should be better to review, now.

@flavorjones Could you please review this change?

I just rebased this PR to master.

This looks good to me. Merging.


Enable interactive mode

If concoct is executed with no arguments, it should start in interactive mode similar to Python. Users can evaluate expressions and run statements within the interactive prompt if desired.

Added in c15d934a.